Sep 17 23:04:23.158627 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:04:23.158644 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:04:23.158654 ip-10-0-139-248 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:04:23.159028 ip-10-0-139-248 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:04:33.373572 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 23:04:33.373591 ip-10-0-139-248 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 665c2478f3cb4a8fb41bf4612ef4ce17 --
Sep 17 23:06:29.470139 ip-10-0-139-248 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:06:29.873399 ip-10-0-139-248 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:06:29.873399 ip-10-0-139-248 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:06:29.873399 ip-10-0-139-248 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:06:29.873399 ip-10-0-139-248 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:06:29.873399 ip-10-0-139-248 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:06:29.876152 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.876062    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:06:29.880289 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880267    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:29.880289 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880285    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:29.880289 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880290    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:29.880289 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880294    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880299    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880304    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880308    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880312    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880316    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880323    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880330    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880334    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880338    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880343    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880347    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880351    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880354    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880359    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880363    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880367    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880371    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880375    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:29.880520 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880379    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880383    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880387    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880391    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880395    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880399    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880403    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880407    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880413    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880417    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880421    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880425    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880430    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880434    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880439    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880445    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880452    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880458    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880463    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880468    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:29.881367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880473    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880477    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880482    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880487    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880491    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880496    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880501    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880505    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880509    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880513    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880516    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880520    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880525    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880528    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880532    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880537    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880541    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880546    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880550    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880554    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:29.882242 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880560    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880564    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880568    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880573    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880577    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880581    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880586    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880590    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880595    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880600    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880605    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880609    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880614    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880618    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880622    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880627    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880632    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880637    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880643    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:29.883062 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880649    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880654    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880659    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880664    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880668    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880675    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880679    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880684    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880688    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880692    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880696    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880700    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880705    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880709    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880713    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880719    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880723    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.880728    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881911    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881923    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:29.883878 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881929    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881934    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881939    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881944    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881949    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881953    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881957    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881961    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881966    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881970    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881974    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881978    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881982    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881987    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881991    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881995    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.881999    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882003    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882009    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882013    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:29.884676 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882018    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882022    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882027    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882031    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882035    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882040    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882044    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882048    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882054    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882058    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882063    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882068    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882072    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882076    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882080    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882084    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882088    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882092    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882096    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:29.885577 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882100    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882104    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882108    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882112    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882116    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882121    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882125    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882129    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882133    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882137    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882142    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882146    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882150    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882154    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882158    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882162    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882166    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882171    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882175    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882179    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:29.886360 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882183    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882187    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882192    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882197    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882201    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882205    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882209    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882213    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882218    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882222    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882226    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882230    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882234    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882238    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882242    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882246    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882251    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882255    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882259    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882262    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:29.886909 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882267    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882271    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882275    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882279    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882283    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882287    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882293    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882299    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882304    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882311    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882316    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882321    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882326    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882330    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882335    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882340    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882345    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.882350    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884134    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:29.887481 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884149    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884154    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884162    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884169    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884176    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884180    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884186    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884190    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884195    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884200    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884205    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884209    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884213    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884217    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884222    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884226    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884231    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884235    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884239    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884244    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:29.887994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884248    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884252    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884257    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884261    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884265    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884270    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884274    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884278    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884283    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884287    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884292    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884297    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884301    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884305    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884309    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884313    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884317    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884321    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884325    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884333    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:29.888513 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884337    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884341    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884345    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884349    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884353    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884357    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884362    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884366    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884370    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884374    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884378    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884383    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884386    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884390    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884395    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884400    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884405    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884409    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884413    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:29.889203 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884418    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884422    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884427    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884431    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884436    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884440    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884444    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884448    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884454    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884460    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884465    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884469    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884473    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884477    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884482    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884486    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884490    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884494    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884498    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884502    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:29.889691 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884506    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884510    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884516    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884522    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884527    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884531    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884535    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884539    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884544    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884548    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884553    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884556    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884560    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884565    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884571    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884578    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884582    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884587    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.884591    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884724    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:06:29.890237 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884772    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884783    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884789    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884819    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884825    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884832    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884838    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884844    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884848    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884854    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884859    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884864    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884869    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884873    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884878    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884882    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884888    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884895    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884899    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884904    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884910    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884915    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884921    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:06:29.890844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884925    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884930    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884935    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884940    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884945    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884949    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884954    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884959    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884965    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884970    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884974    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884979    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884983    2580 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884988    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.884995    2580 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885000    2580 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885004    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885008    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885013    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885019    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885024    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885028    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885033    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885037    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885042    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:06:29.891401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885046    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885051    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885058    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885063    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885068    2580 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885073    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885078    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885083    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885088    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885093    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885098    2580 flags.go:64] FLAG: --help="false"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885103    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-139-248.ec2.internal"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885108    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885113    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885117    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885123    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885128    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885133    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885137    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885142    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885146    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885151    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885155    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885160    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:06:29.892059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885165    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885169    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885174    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885179    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885184    2580 flags.go:64] FLAG: --lock-file=""
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885188    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885193    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885197    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885206    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885211    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885217    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885222    2580 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885227    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885232    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885236    2580 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885241    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885248    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885252    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885258    2580 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885263    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885268    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885273    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885277    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885282    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885287    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:06:29.892653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885292    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885304    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885309    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885314    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885318    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885323    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885332    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885336    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885341    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885345    2580 flags.go:64] FLAG: --port="10250"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885350    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885355    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ed7d5e59e12828d1"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885362    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885366    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885371    2580 flags.go:64] FLAG: --register-node="true"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885375    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885381    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885388    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885394    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885399    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885405    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885410    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885415    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885420    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885425    2580 flags.go:64] FLAG: --runonce="false"
Sep 17 23:06:29.893286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885429    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885434    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885439    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885444    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885449    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885453    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885458    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885463    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885468    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885472    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885477    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885481    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885486    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885491    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885508    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885519    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885524    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885529    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885535    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885543    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885548    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885552    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885557    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885563    2580 flags.go:64] FLAG: --v="2"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885571    2580 flags.go:64] FLAG: --version="false"
Sep 17 23:06:29.894011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885577    2580 flags.go:64] FLAG: --vmodule=""
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885583    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.885588    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885764    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885771    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885776    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885782    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885786    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885791    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885815    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885820    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885824    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885829    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885833    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885837    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885841    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885845    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885849    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885854    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885858    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:29.894743 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885862    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885866    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885870    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885874    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885878    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885882    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885889    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885893    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885897    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885901    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885905    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885912    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885917    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885921    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885927    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885931    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885935    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885942    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885947    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:29.895271 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885951    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885956    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885960    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885964    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885968    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885972    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885977    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885981    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885985    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885990    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885994    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.885998    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886002    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886006    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886010    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886014    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886018    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886022    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886026    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886032    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:29.895751 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886036    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886041    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886044    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886048    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886056    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886061    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886068    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886072    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886077    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886081    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886085    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886089    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886093    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886097    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886101    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886105    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886109    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886113    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886117    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886121    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:29.896253 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886125    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886129    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886134    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886140    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886145    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886151    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886156    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886161    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886165    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886169    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886173    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886179    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886184    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886188    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886192    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886196    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886201    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886206    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886210    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:29.896744 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886214    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886218    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886224    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.886228    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.887158    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.895113    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.895130    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.895149    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.895158    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895212    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895217    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895221    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895225    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895228    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:29.897264 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895231    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895234    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895237    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895240    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895243    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895246    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895249    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895252    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895255    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895257    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895260    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895263    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895266    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895269    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895272    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895274    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895277    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895280    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895283    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895285    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:29.897610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895288    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895291    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895293    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895296    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895300    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895303    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895305    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895308    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895312    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895316    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895319    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895321    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895324    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895326    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895328    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895331    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895334    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895337    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895339    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895342    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:29.898111 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895344    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895347    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895349    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895352    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895355    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895358    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895361    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895363    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895367    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895371    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895375    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895379    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895384    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895388    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895392    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895395    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895399    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895403    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895407    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:29.898616 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895411    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895414    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895418    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895421    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895425    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895428    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895434    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895439    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895443    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895447    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895451    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895455    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895459    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895463    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895466    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895470    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895473    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895477    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895480    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:29.899114 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895484    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895488    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895493    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895497    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895501    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895505    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895509    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895512    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895516    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895520    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895524    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895528    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895533    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895537    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895541    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:29.899569 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895545    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.895553    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895710    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895715    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895719    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895722    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895725    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895728    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895731    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895734    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895737    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895740    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895743    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895745    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895748    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:29.899954 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895751    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895755    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895759    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895762    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895765    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895768    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895770    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895773    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895776    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895778    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895780    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895783    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895786    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895789    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895791    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895795    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895811    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895814    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895816    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:29.900336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895819    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895822    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895824    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895827    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895829    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895832    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895834    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895837    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895839    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895842    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895844    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895847    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895849    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895852    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895855    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895857    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895859    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895862    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895865    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895867    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:29.900851 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895870    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895872    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895875    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895878    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895880    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895883    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895885    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895888    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895890    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895894    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895898    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895902    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895905    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895908    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895910    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895913    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895916    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895919    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895921    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:29.901335 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895924    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895926    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895929    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895931    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895934    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895936    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895939    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895941    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895944    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895946    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895949    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895951    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895953    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895956    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895958    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895961    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895963    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895966    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895968    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895971    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:29.901788 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895974    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895976    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895979    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895982    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895985    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895987    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895989    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:29.895993    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.895998    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.896119    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.898565    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.899590    2580 server.go:1019] "Starting client certificate rotation"
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.899691    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:06:29.902286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.899727    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:06:29.902619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.902319    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:06:29.904452 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.904426    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:06:29.916303 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.916283    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:06:29.918719 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.918697    2580 log.go:25] "Validated CRI v1 image API"
Sep 17 23:06:29.920359 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.920343    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:06:29.922952 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.922931    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e29bb3f0-8872-4dc3-b6af-815a537940ac:/dev/nvme0n1p4 f4b22312-8f9a-45b4-9d32-314ada03358a:/dev/nvme0n1p3]
Sep 17 23:06:29.923007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.922952    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:06:29.928326 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.928307    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:06:29.929504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.929397    2580 manager.go:216] Machine: {Timestamp:2026-09-17 23:06:29.927639689 +0000 UTC m=+0.357499517 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090481 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29888124f86e5314f437dc8e73ba70 SystemUUID:ec298881-24f8-6e53-14f4-37dc8e73ba70 BootID:665c2478-f3cb-4a8f-b41b-f4612ef4ce17 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:4f:89:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:4f:89:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:48:bf:bc:e0:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:06:29.929504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.929500    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:06:29.929621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.929583    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:06:29.933612 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.933587    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:06:29.933753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.933615    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:06:29.935092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.935081    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:06:29.935131 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.935094    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:06:29.935131 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.935101    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:06:29.936007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.935996    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:06:29.936084 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.936075    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:06:29.940519 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.940508    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:06:29.940630 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.940621    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:06:29.940662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.940650    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:06:29.940662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.940660    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:06:29.940715 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.940671    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:06:29.940715 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.940681    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:06:29.940715 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.940689    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:06:29.941037 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.941025    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:06:29.941086 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.941045    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:06:29.943333 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.943318    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:06:29.946206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.946193    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:06:29.946247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.946223    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:06:29.947639 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947625    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:06:29.947704 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947645    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:06:29.947704 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947652    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:06:29.947704 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947659    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:06:29.947704 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947673    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:06:29.947704 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947685    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:06:29.947704 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947701    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:06:29.947704 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947707    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:06:29.947931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947714    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:06:29.947931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947719    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:06:29.947931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947727    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:06:29.947931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.947732    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:06:29.948506 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.948496    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:06:29.948542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.948509    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:06:29.950532 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.950511    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:06:29.951366 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:29.951337    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:06:29.951411 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:29.951332    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:06:29.952555 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.952545    2580 server.go:1268] "Started kubelet"
Sep 17 23:06:29.952778 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.952720    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:06:29.952861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.952773    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:06:29.952907 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.952891    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:06:29.953508 ip-10-0-139-248 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:06:29.955015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.954973    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:06:29.955748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.955720    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-krmpq"
Sep 17 23:06:29.959717 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.959659    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:06:29.962093 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:29.962056    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:06:29.962206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.962190    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-krmpq"
Sep 17 23:06:29.966070 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.966049    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:06:29.966750 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.966727    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:06:29.967015 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:29.966961    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:29.967439 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.967422    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:06:29.967490 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.967424    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:06:29.967790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.967450    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:06:29.967871 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.967544    2580 factory.go:55] Registering systemd factory
Sep 17 23:06:29.967871 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.967859    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:06:29.967954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.967893    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:06:29.967954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.967903    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:06:29.968111 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.968094    2580 factory.go:153] Registering CRI-O factory
Sep 17 23:06:29.968111 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.968112    2580 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:06:29.968254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.968162    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:06:29.968254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.968181    2580 factory.go:103] Registering Raw factory
Sep 17 23:06:29.968254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.968196    2580 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:06:29.968652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.968638    2580 manager.go:349] Starting recovery of all containers
Sep 17 23:06:29.968840 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:29.968791    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:06:29.968946 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:29.968886    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:06:29.970107 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:29.969000    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-248.ec2.internal.18d63e4a1c7890cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-248.ec2.internal,UID:ip-10-0-139-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-248.ec2.internal,},FirstTimestamp:2026-09-17 23:06:29.952516299 +0000 UTC m=+0.382376110,LastTimestamp:2026-09-17 23:06:29.952516299 +0000 UTC m=+0.382376110,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-248.ec2.internal,}"
Sep 17 23:06:29.979480 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.979460    2580 manager.go:354] Recovery completed
Sep 17 23:06:29.983678 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.983666    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:29.985565 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.985546    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:29.985626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.985574    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:29.985626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.985587    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:29.986235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.986210    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:06:29.986235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.986225    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:29.986329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.986228    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:06:29.986329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.986324    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:06:29.986920 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.986904    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:29.986983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.986937    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:29.986983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.986953    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:29.988149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.988136    2580 policy_none.go:49] "None policy: Start"
Sep 17 23:06:29.988193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.988153    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:06:29.988193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.988163    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:06:29.990466 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:29.990453    2580 policy_none.go:47] "Start"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.038331    2580 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.038368    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.038378    2580 server.go:87] "Starting device plugin registration server"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.038762    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.038774    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.038916    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.039005    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.039015    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.039490    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:06:30.046749 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.039538    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.099916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.099873    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:06:30.101336 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.101317    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:06:30.101402 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.101379    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:06:30.101449 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.101412    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:06:30.101495 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.101459    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:06:30.104047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.104018    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:30.139297 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.139274    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:30.140255 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.140239    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:30.140317 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.140270    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:30.140317 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.140280    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:30.140317 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.140309    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.146938 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.146923    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.146985 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.146946    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-248.ec2.internal\": node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.170056 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.170029    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.202152 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.202124    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"]
Sep 17 23:06:30.202230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.202216    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:30.203277 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.203261    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:30.203342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.203291    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:30.203342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.203304    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:30.205768 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.205756    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:30.205887 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.205873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.205936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.205906    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:30.207969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.207953    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:30.207969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.207964    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:30.208115 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.207985    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:30.208115 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.207986    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:30.208115 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.207994    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:30.208115 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.208001    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:30.210362 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.210342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.210460 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.210368    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:30.213698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.213683    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:30.213772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.213710    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:30.213772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.213722    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:30.227414 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.227393    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.ec2.internal\" not found" node="ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.233166 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.233150    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.ec2.internal\" not found" node="ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.269391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.269356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21013831982d43c783952b1c4eba40a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"21013831982d43c783952b1c4eba40a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.269391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.269395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/911b0f78153f44b2f6303a5e127feea0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"911b0f78153f44b2f6303a5e127feea0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.269391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.269413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/911b0f78153f44b2f6303a5e127feea0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"911b0f78153f44b2f6303a5e127feea0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.270398 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.270380    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.369891 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.369854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/911b0f78153f44b2f6303a5e127feea0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"911b0f78153f44b2f6303a5e127feea0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.369891 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.369889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/911b0f78153f44b2f6303a5e127feea0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"911b0f78153f44b2f6303a5e127feea0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.370069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.369907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21013831982d43c783952b1c4eba40a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"21013831982d43c783952b1c4eba40a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.370069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.369966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/911b0f78153f44b2f6303a5e127feea0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"911b0f78153f44b2f6303a5e127feea0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.370069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.369968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/911b0f78153f44b2f6303a5e127feea0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"911b0f78153f44b2f6303a5e127feea0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.370069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.370027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21013831982d43c783952b1c4eba40a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"21013831982d43c783952b1c4eba40a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.370871 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.370856    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.471723 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.471639    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.530939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.530908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.537223 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.537204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.572768 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.572733    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.673220 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.673186    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.773893 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.773811    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
Sep 17 23:06:30.811775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.811748    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:30.867755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.867724    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.878547 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.878514    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:06:30.880435 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.880420    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
Sep 17 23:06:30.888443 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.888428    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:06:30.899583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.899557    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:06:30.899706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.899690    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:06:30.899749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.899725    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:06:30.940887 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.940853    2580 apiserver.go:52] "Watching apiserver"
Sep 17 23:06:30.955190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.955160    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:06:30.956043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.956017    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal","openshift-multus/multus-additional-cni-plugins-rzs4g","openshift-multus/multus-vvrj6","openshift-network-diagnostics/network-check-target-xrmrc","openshift-network-operator/iptables-alerter-mzww5","kube-system/global-pull-secret-syncer-zlv7c","kube-system/konnectivity-agent-498dh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk","openshift-cluster-node-tuning-operator/tuned-sk49c","openshift-dns/node-resolver-6rzcj","openshift-image-registry/node-ca-hv4vh","openshift-multus/network-metrics-daemon-njwnq","openshift-ovn-kubernetes/ovnkube-node-rfdm4","kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"]
Sep 17 23:06:30.958750 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.958726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:30.960936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.960919    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:06:30.961043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.960999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4kjgr\""
Sep 17 23:06:30.961043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.961006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:06:30.961251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.961236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.963538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.963423    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:06:30.963538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.963518    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:06:30.963790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.963772    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:06:30.964130 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.964113    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:06:30.964208 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.964171    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-77wxj\""
Sep 17 23:06:30.964585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.964569    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:06:30.964713 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.964583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.965172 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.965150    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:06:30.966205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.966187    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:06:30.966573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.966557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:06:30.966652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.966564    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wv5r5\""
Sep 17 23:06:30.966790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.966775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:06:30.966894 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.966879    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:06:30.966999 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.966986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:06:30.967524 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.967510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:30.967596 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.967579    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:30.969620 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.969587    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:01:30 +0000 UTC" deadline="2028-07-03 21:30:35.54429487 +0000 UTC"
Sep 17 23:06:30.969620 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.969619    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15718h24m4.574677927s"
Sep 17 23:06:30.970092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.970081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:30.971949 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.971925    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:06:30.971949 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.971943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pt6wz\""
Sep 17 23:06:30.972083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.971999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:06:30.972083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:06:30.972668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-var-lib-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.972748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-ovn\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.972748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-cni-bin\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.972748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:30.972748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97c14bc-f869-4487-b479-4576d1dd4493-ovn-node-metrics-cert\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-cnibin\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-hostroot\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-conf-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.972813    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-node-log\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-log-socket\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-ovnkube-script-lib\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.972918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-daemon-config\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8bf\" (UniqueName: \"kubernetes.io/projected/688a3748-ca64-4527-a885-37b44ede19ec-kube-api-access-wt8bf\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-systemd\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-etc-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.972991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-run-ovn-kubernetes\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9prw\" (UniqueName: \"kubernetes.io/projected/d97c14bc-f869-4487-b479-4576d1dd4493-kube-api-access-q9prw\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-kubelet\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-run-netns\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-ovnkube-config\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-env-overrides\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-netns\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/688a3748-ca64-4527-a885-37b44ede19ec-iptables-alerter-script\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:30.973193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-system-cni-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-cni-bin\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-systemd-units\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-slash\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-socket-dir-parent\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-k8s-cni-cncf-io\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-cni-multus\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-etc-kubernetes\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56jx\" (UniqueName: \"kubernetes.io/projected/5ed5a1fa-247e-4efb-b57f-39fd524d7981-kube-api-access-x56jx\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/217ae11d-95f6-4818-831d-700279f0fd51-agent-certs\") pod \"konnectivity-agent-498dh\" (UID: \"217ae11d-95f6-4818-831d-700279f0fd51\") " pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/217ae11d-95f6-4818-831d-700279f0fd51-konnectivity-ca\") pod \"konnectivity-agent-498dh\" (UID: \"217ae11d-95f6-4818-831d-700279f0fd51\") " pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-cni-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-kubelet\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-multus-certs\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688a3748-ca64-4527-a885-37b44ede19ec-host-slash\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-cni-netd\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:30.973673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.973534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ed5a1fa-247e-4efb-b57f-39fd524d7981-cni-binary-copy\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:30.975536 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.975516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:30.977885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.977869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:06:30.978037 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.977869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:06:30.978194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.978169    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-52fs4\""
Sep 17 23:06:30.978292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.978277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:30.978682 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.978661    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:06:30.980182 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.980165    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:06:30.980327 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.980311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:06:30.980395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.980331    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hqpvp\""
Sep 17 23:06:30.980395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.980377    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:06:30.981624 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.981605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:30.983570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.983555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dx72d\""
Sep 17 23:06:30.983677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.983661    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:06:30.983778 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.983694    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:06:30.984764 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.984750    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:30.986560 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.986542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:06:30.986648 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.986566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:30.986705 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.986662    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:06:30.986705 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.986672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hqrmh\""
Sep 17 23:06:30.988675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.988659    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:06:30.988753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.988733    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:06:30.988886 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.988871    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cr9qc\""
Sep 17 23:06:30.988934 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.988871    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:06:30.989082 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.989066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:30.989160 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:30.989139    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:30.998533 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:30.998497    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bqdlf"
Sep 17 23:06:31.007005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.006979    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bqdlf"
Sep 17 23:06:31.008898 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.008870    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:31.068090 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.068065    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:06:31.074063 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86t9m\" (UniqueName: \"kubernetes.io/projected/8896e89b-3c26-4006-95f0-5620d676cb48-kube-api-access-86t9m\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.074197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-run\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.074197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8f6\" (UniqueName: \"kubernetes.io/projected/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-kube-api-access-6j8f6\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.074197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-ovnkube-script-lib\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.074197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8bf\" (UniqueName: \"kubernetes.io/projected/688a3748-ca64-4527-a885-37b44ede19ec-kube-api-access-wt8bf\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:31.074197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.074197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-sys-fs\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.074197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-socket-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-device-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8d4v\" (UniqueName: \"kubernetes.io/projected/170daf82-cc47-44fb-a6d9-141059340583-kube-api-access-p8d4v\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-netns\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-cni-bin\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-netns\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.074444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-cni-bin\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-modprobe-d\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysctl-d\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-lib-modules\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-tuned\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-slash\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56jx\" (UniqueName: \"kubernetes.io/projected/5ed5a1fa-247e-4efb-b57f-39fd524d7981-kube-api-access-x56jx\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-host\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-ovnkube-script-lib\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnb6\" (UniqueName: \"kubernetes.io/projected/4cf297b0-5433-4350-879a-8447d2e1d39c-kube-api-access-gfnb6\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.074777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-slash\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-registration-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cnibin\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmr6\" (UniqueName: \"kubernetes.io/projected/cefaeeee-fef2-4a38-a4a1-d26e2b452228-kube-api-access-zjmr6\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/217ae11d-95f6-4818-831d-700279f0fd51-konnectivity-ca\") pod \"konnectivity-agent-498dh\" (UID: \"217ae11d-95f6-4818-831d-700279f0fd51\") " pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-cni-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-kubelet\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.074975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-hosts-file\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ed5a1fa-247e-4efb-b57f-39fd524d7981-cni-binary-copy\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysctl-conf\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-cni-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-kubelet\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-tmp-dir\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-var-lib-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-hostroot\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-var-lib-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-node-log\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-hostroot\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-log-socket\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-daemon-config\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-log-socket\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-node-log\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8896e89b-3c26-4006-95f0-5620d676cb48-host\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-etc-selinux\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/217ae11d-95f6-4818-831d-700279f0fd51-konnectivity-ca\") pod \"konnectivity-agent-498dh\" (UID: \"217ae11d-95f6-4818-831d-700279f0fd51\") " pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26053f12-63eb-48b5-a610-95ce5cd2c941-dbus\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ed5a1fa-247e-4efb-b57f-39fd524d7981-cni-binary-copy\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-systemd\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-etc-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-systemd\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-etc-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-openvswitch\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-run-ovn-kubernetes\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.075988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-run-ovn-kubernetes\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9prw\" (UniqueName: \"kubernetes.io/projected/d97c14bc-f869-4487-b479-4576d1dd4493-kube-api-access-q9prw\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-sys\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-var-lib-kubelet\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-kubelet\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-run-netns\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-ovnkube-config\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-daemon-config\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-env-overrides\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-run-netns\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-kubelet\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/688a3748-ca64-4527-a885-37b44ede19ec-iptables-alerter-script\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-system-cni-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.075984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6dl\" (UniqueName: \"kubernetes.io/projected/f6a51291-55ed-41ea-8029-bf5b677956a8-kube-api-access-jr6dl\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.076577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-systemd-units\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-socket-dir-parent\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-k8s-cni-cncf-io\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-cni-multus\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-system-cni-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-etc-kubernetes\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-systemd\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-etc-kubernetes\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26053f12-63eb-48b5-a610-95ce5cd2c941-kubelet-config\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/217ae11d-95f6-4818-831d-700279f0fd51-agent-certs\") pod \"konnectivity-agent-498dh\" (UID: \"217ae11d-95f6-4818-831d-700279f0fd51\") " pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-multus-certs\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-socket-dir-parent\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-env-overrides\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d97c14bc-f869-4487-b479-4576d1dd4493-ovnkube-config\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-systemd-units\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-multus-certs\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-run-k8s-cni-cncf-io\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-host-var-lib-cni-multus\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.077201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688a3748-ca64-4527-a885-37b44ede19ec-host-slash\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/688a3748-ca64-4527-a885-37b44ede19ec-host-slash\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8896e89b-3c26-4006-95f0-5620d676cb48-serviceca\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysconfig\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/688a3748-ca64-4527-a885-37b44ede19ec-iptables-alerter-script\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-kubernetes\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-system-cni-dir\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-cni-netd\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf297b0-5433-4350-879a-8447d2e1d39c-tmp\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-cni-netd\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-ovn\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-cni-bin\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-run-ovn\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97c14bc-f869-4487-b479-4576d1dd4493-ovn-node-metrics-cert\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d97c14bc-f869-4487-b479-4576d1dd4493-host-cni-bin\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.077836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-cnibin\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.078543 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-cnibin\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.078543 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-conf-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.078543 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ed5a1fa-247e-4efb-b57f-39fd524d7981-multus-conf-dir\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.078543 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.076901    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:06:31.080426 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.080361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d97c14bc-f869-4487-b479-4576d1dd4493-ovn-node-metrics-cert\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.080629 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.080610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/217ae11d-95f6-4818-831d-700279f0fd51-agent-certs\") pod \"konnectivity-agent-498dh\" (UID: \"217ae11d-95f6-4818-831d-700279f0fd51\") " pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:31.081307 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.081291    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:31.081399 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.081315    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:31.081399 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.081326    2580 projected.go:196] Error preparing data for projected volume kube-api-access-j6dnd for pod openshift-network-diagnostics/network-check-target-xrmrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:31.081399 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.081389    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd podName:8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e nodeName:}" failed. No retries permitted until 2026-09-17 23:06:31.581370743 +0000 UTC m=+2.011230561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j6dnd" (UniqueName: "kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd") pod "network-check-target-xrmrc" (UID: "8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:31.082488 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.082459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8bf\" (UniqueName: \"kubernetes.io/projected/688a3748-ca64-4527-a885-37b44ede19ec-kube-api-access-wt8bf\") pod \"iptables-alerter-mzww5\" (UID: \"688a3748-ca64-4527-a885-37b44ede19ec\") " pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:31.083020 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.082997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56jx\" (UniqueName: \"kubernetes.io/projected/5ed5a1fa-247e-4efb-b57f-39fd524d7981-kube-api-access-x56jx\") pod \"multus-vvrj6\" (UID: \"5ed5a1fa-247e-4efb-b57f-39fd524d7981\") " pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.083951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.083934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9prw\" (UniqueName: \"kubernetes.io/projected/d97c14bc-f869-4487-b479-4576d1dd4493-kube-api-access-q9prw\") pod \"ovnkube-node-rfdm4\" (UID: \"d97c14bc-f869-4487-b479-4576d1dd4493\") " pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.170269 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.170223    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21013831982d43c783952b1c4eba40a7.slice/crio-17018894e62c5a067df6a4482e2c0b33bf8fc24cb328b19fbdb66a779765807d WatchSource:0}: Error finding container 17018894e62c5a067df6a4482e2c0b33bf8fc24cb328b19fbdb66a779765807d: Status 404 returned error can't find the container with id 17018894e62c5a067df6a4482e2c0b33bf8fc24cb328b19fbdb66a779765807d
Sep 17 23:06:31.171249 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.171229    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911b0f78153f44b2f6303a5e127feea0.slice/crio-95a8063a6d27492bf132bbd5e6f550276fcfe896f44d8917995a16dcdc9f05da WatchSource:0}: Error finding container 95a8063a6d27492bf132bbd5e6f550276fcfe896f44d8917995a16dcdc9f05da: Status 404 returned error can't find the container with id 95a8063a6d27492bf132bbd5e6f550276fcfe896f44d8917995a16dcdc9f05da
Sep 17 23:06:31.175385 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.175370    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:06:31.177129 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-host\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnb6\" (UniqueName: \"kubernetes.io/projected/4cf297b0-5433-4350-879a-8447d2e1d39c-kube-api-access-gfnb6\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-registration-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.177230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cnibin\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.177230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmr6\" (UniqueName: \"kubernetes.io/projected/cefaeeee-fef2-4a38-a4a1-d26e2b452228-kube-api-access-zjmr6\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.177230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-host\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-hosts-file\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-registration-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysctl-conf\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-hosts-file\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-tmp-dir\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysctl-conf\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8896e89b-3c26-4006-95f0-5620d676cb48-host\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8896e89b-3c26-4006-95f0-5620d676cb48-host\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cnibin\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-etc-selinux\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.177459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26053f12-63eb-48b5-a610-95ce5cd2c941-dbus\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-sys\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-var-lib-kubelet\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-etc-selinux\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/26053f12-63eb-48b5-a610-95ce5cd2c941-dbus\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6dl\" (UniqueName: \"kubernetes.io/projected/f6a51291-55ed-41ea-8029-bf5b677956a8-kube-api-access-jr6dl\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-sys\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-var-lib-kubelet\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-systemd\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26053f12-63eb-48b5-a610-95ce5cd2c941-kubelet-config\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.177686    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8896e89b-3c26-4006-95f0-5620d676cb48-serviceca\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysconfig\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.177757    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs podName:f6a51291-55ed-41ea-8029-bf5b677956a8 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:31.677737709 +0000 UTC m=+2.107597529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs") pod "network-metrics-daemon-njwnq" (UID: "f6a51291-55ed-41ea-8029-bf5b677956a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysconfig\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.177944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-kubernetes\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-systemd\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-system-cni-dir\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf297b0-5433-4350-879a-8447d2e1d39c-tmp\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86t9m\" (UniqueName: \"kubernetes.io/projected/8896e89b-3c26-4006-95f0-5620d676cb48-kube-api-access-86t9m\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-run\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.177996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8f6\" (UniqueName: \"kubernetes.io/projected/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-kube-api-access-6j8f6\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-sys-fs\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-socket-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-device-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8d4v\" (UniqueName: \"kubernetes.io/projected/170daf82-cc47-44fb-a6d9-141059340583-kube-api-access-p8d4v\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-modprobe-d\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysctl-d\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.178820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-lib-modules\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-tuned\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8896e89b-3c26-4006-95f0-5620d676cb48-serviceca\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-kubernetes\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cefaeeee-fef2-4a38-a4a1-d26e2b452228-system-cni-dir\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-tmp-dir\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-socket-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178408    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/26053f12-63eb-48b5-a610-95ce5cd2c941-kubelet-config\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.178405    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.178467    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret podName:26053f12-63eb-48b5-a610-95ce5cd2c941 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:31.678450986 +0000 UTC m=+2.108310783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret") pod "global-pull-secret-syncer-zlv7c" (UID: "26053f12-63eb-48b5-a610-95ce5cd2c941") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-run\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-device-dir\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-modprobe-d\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.179759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/170daf82-cc47-44fb-a6d9-141059340583-sys-fs\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.180663 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-sysctl-d\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.180663 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf297b0-5433-4350-879a-8447d2e1d39c-lib-modules\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.180663 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.178996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cefaeeee-fef2-4a38-a4a1-d26e2b452228-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.180969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.180951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf297b0-5433-4350-879a-8447d2e1d39c-etc-tuned\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.181064 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.181048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf297b0-5433-4350-879a-8447d2e1d39c-tmp\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.185163 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.185135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6dl\" (UniqueName: \"kubernetes.io/projected/f6a51291-55ed-41ea-8029-bf5b677956a8-kube-api-access-jr6dl\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:31.185441 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.185425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmr6\" (UniqueName: \"kubernetes.io/projected/cefaeeee-fef2-4a38-a4a1-d26e2b452228-kube-api-access-zjmr6\") pod \"multus-additional-cni-plugins-rzs4g\" (UID: \"cefaeeee-fef2-4a38-a4a1-d26e2b452228\") " pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.186644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.186621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnb6\" (UniqueName: \"kubernetes.io/projected/4cf297b0-5433-4350-879a-8447d2e1d39c-kube-api-access-gfnb6\") pod \"tuned-sk49c\" (UID: \"4cf297b0-5433-4350-879a-8447d2e1d39c\") " pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.186843 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.186826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8f6\" (UniqueName: \"kubernetes.io/projected/f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62-kube-api-access-6j8f6\") pod \"node-resolver-6rzcj\" (UID: \"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62\") " pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.187023 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.187004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8d4v\" (UniqueName: \"kubernetes.io/projected/170daf82-cc47-44fb-a6d9-141059340583-kube-api-access-p8d4v\") pod \"aws-ebs-csi-driver-node-mbzhk\" (UID: \"170daf82-cc47-44fb-a6d9-141059340583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.187124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.187108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86t9m\" (UniqueName: \"kubernetes.io/projected/8896e89b-3c26-4006-95f0-5620d676cb48-kube-api-access-86t9m\") pod \"node-ca-hv4vh\" (UID: \"8896e89b-3c26-4006-95f0-5620d676cb48\") " pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.294725 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.294648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:31.304847 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.304824    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217ae11d_95f6_4818_831d_700279f0fd51.slice/crio-4b40c89fa64f4c0c9c586850a8e1f86907cf75800aa381516cb9e15c425d4ae8 WatchSource:0}: Error finding container 4b40c89fa64f4c0c9c586850a8e1f86907cf75800aa381516cb9e15c425d4ae8: Status 404 returned error can't find the container with id 4b40c89fa64f4c0c9c586850a8e1f86907cf75800aa381516cb9e15c425d4ae8
Sep 17 23:06:31.309322 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.309305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:31.317397 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.317375    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97c14bc_f869_4487_b479_4576d1dd4493.slice/crio-d1f715b95c8f77c260bcab7f1fc357920106b0bb327f7c59cd1a3eb8d72b6823 WatchSource:0}: Error finding container d1f715b95c8f77c260bcab7f1fc357920106b0bb327f7c59cd1a3eb8d72b6823: Status 404 returned error can't find the container with id d1f715b95c8f77c260bcab7f1fc357920106b0bb327f7c59cd1a3eb8d72b6823
Sep 17 23:06:31.329461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.329440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vvrj6"
Sep 17 23:06:31.337775 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.337750    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed5a1fa_247e_4efb_b57f_39fd524d7981.slice/crio-13103cb9f5022641f2eed539e72b6faefd0b7e78f85cf5b453226dca7dddd1c3 WatchSource:0}: Error finding container 13103cb9f5022641f2eed539e72b6faefd0b7e78f85cf5b453226dca7dddd1c3: Status 404 returned error can't find the container with id 13103cb9f5022641f2eed539e72b6faefd0b7e78f85cf5b453226dca7dddd1c3
Sep 17 23:06:31.348893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.348875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mzww5"
Sep 17 23:06:31.357141 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.357122    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688a3748_ca64_4527_a885_37b44ede19ec.slice/crio-572380d8cd5fed6986da26c7eb1cd36f55380e8ac1d74ccfe10fb2495b14eba2 WatchSource:0}: Error finding container 572380d8cd5fed6986da26c7eb1cd36f55380e8ac1d74ccfe10fb2495b14eba2: Status 404 returned error can't find the container with id 572380d8cd5fed6986da26c7eb1cd36f55380e8ac1d74ccfe10fb2495b14eba2
Sep 17 23:06:31.364581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.364560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sk49c"
Sep 17 23:06:31.372674 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.372646    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf297b0_5433_4350_879a_8447d2e1d39c.slice/crio-59b96c61b4d71f47ddac36a05c87ec1fbaf1ff317f630c778fbb707fd1c70487 WatchSource:0}: Error finding container 59b96c61b4d71f47ddac36a05c87ec1fbaf1ff317f630c778fbb707fd1c70487: Status 404 returned error can't find the container with id 59b96c61b4d71f47ddac36a05c87ec1fbaf1ff317f630c778fbb707fd1c70487
Sep 17 23:06:31.377963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.377947    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk"
Sep 17 23:06:31.385838 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.385811    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170daf82_cc47_44fb_a6d9_141059340583.slice/crio-524e3cb9b96f1475a566d5367345fbd9817130735289db4469cac3b23a9384a8 WatchSource:0}: Error finding container 524e3cb9b96f1475a566d5367345fbd9817130735289db4469cac3b23a9384a8: Status 404 returned error can't find the container with id 524e3cb9b96f1475a566d5367345fbd9817130735289db4469cac3b23a9384a8
Sep 17 23:06:31.392623 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.392604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzs4g"
Sep 17 23:06:31.401457 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.401433    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefaeeee_fef2_4a38_a4a1_d26e2b452228.slice/crio-77a20a9f5a91e1549d7371520e7042e305100f89461e895eb41335a34aa567e4 WatchSource:0}: Error finding container 77a20a9f5a91e1549d7371520e7042e305100f89461e895eb41335a34aa567e4: Status 404 returned error can't find the container with id 77a20a9f5a91e1549d7371520e7042e305100f89461e895eb41335a34aa567e4
Sep 17 23:06:31.407183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.407163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6rzcj"
Sep 17 23:06:31.411822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.411779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hv4vh"
Sep 17 23:06:31.417151 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.417131    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fc1ca6_8c87_471a_bc2b_bb5f9fbb4d62.slice/crio-66ff8eb38ee5dd467ec05e590b2c71ca3f5a01e7b5660437f37fb97799e64814 WatchSource:0}: Error finding container 66ff8eb38ee5dd467ec05e590b2c71ca3f5a01e7b5660437f37fb97799e64814: Status 404 returned error can't find the container with id 66ff8eb38ee5dd467ec05e590b2c71ca3f5a01e7b5660437f37fb97799e64814
Sep 17 23:06:31.421042 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:06:31.421018    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8896e89b_3c26_4006_95f0_5620d676cb48.slice/crio-4f42da47bf0c496588eaa0e0a7c5ae592362e4f57c62c8252a63e88252da0ae4 WatchSource:0}: Error finding container 4f42da47bf0c496588eaa0e0a7c5ae592362e4f57c62c8252a63e88252da0ae4: Status 404 returned error can't find the container with id 4f42da47bf0c496588eaa0e0a7c5ae592362e4f57c62c8252a63e88252da0ae4
Sep 17 23:06:31.526429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.526403    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:31.581575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.581484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:31.581724 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.581703    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:31.581773 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.581735    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:31.581773 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.581747    2580 projected.go:196] Error preparing data for projected volume kube-api-access-j6dnd for pod openshift-network-diagnostics/network-check-target-xrmrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:31.581859 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.581834    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd podName:8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e nodeName:}" failed. No retries permitted until 2026-09-17 23:06:32.581815937 +0000 UTC m=+3.011675746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6dnd" (UniqueName: "kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd") pod "network-check-target-xrmrc" (UID: "8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:31.682671 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.682640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:31.682881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:31.682741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:31.682881 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.682824    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:31.682881 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.682849    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:31.683044 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.682894    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs podName:f6a51291-55ed-41ea-8029-bf5b677956a8 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:32.682877315 +0000 UTC m=+3.112737112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs") pod "network-metrics-daemon-njwnq" (UID: "f6a51291-55ed-41ea-8029-bf5b677956a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:31.683044 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:31.682909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret podName:26053f12-63eb-48b5-a610-95ce5cd2c941 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:32.682901466 +0000 UTC m=+3.112761263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret") pod "global-pull-secret-syncer-zlv7c" (UID: "26053f12-63eb-48b5-a610-95ce5cd2c941") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:32.010545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.010296    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:31 +0000 UTC" deadline="2028-04-21 21:14:35.15723838 +0000 UTC"
Sep 17 23:06:32.010545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.010366    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13966h8m3.146877011s"
Sep 17 23:06:32.148839 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.148206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk" event={"ID":"170daf82-cc47-44fb-a6d9-141059340583","Type":"ContainerStarted","Data":"524e3cb9b96f1475a566d5367345fbd9817130735289db4469cac3b23a9384a8"}
Sep 17 23:06:32.161336 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.161276    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:32.162638 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.162550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sk49c" event={"ID":"4cf297b0-5433-4350-879a-8447d2e1d39c","Type":"ContainerStarted","Data":"59b96c61b4d71f47ddac36a05c87ec1fbaf1ff317f630c778fbb707fd1c70487"}
Sep 17 23:06:32.168828 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.168732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"d1f715b95c8f77c260bcab7f1fc357920106b0bb327f7c59cd1a3eb8d72b6823"}
Sep 17 23:06:32.177932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.177890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" event={"ID":"21013831982d43c783952b1c4eba40a7","Type":"ContainerStarted","Data":"17018894e62c5a067df6a4482e2c0b33bf8fc24cb328b19fbdb66a779765807d"}
Sep 17 23:06:32.209044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.208997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6rzcj" event={"ID":"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62","Type":"ContainerStarted","Data":"66ff8eb38ee5dd467ec05e590b2c71ca3f5a01e7b5660437f37fb97799e64814"}
Sep 17 23:06:32.228369 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.228309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerStarted","Data":"77a20a9f5a91e1549d7371520e7042e305100f89461e895eb41335a34aa567e4"}
Sep 17 23:06:32.243097 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.243049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mzww5" event={"ID":"688a3748-ca64-4527-a885-37b44ede19ec","Type":"ContainerStarted","Data":"572380d8cd5fed6986da26c7eb1cd36f55380e8ac1d74ccfe10fb2495b14eba2"}
Sep 17 23:06:32.250998 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.250956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vvrj6" event={"ID":"5ed5a1fa-247e-4efb-b57f-39fd524d7981","Type":"ContainerStarted","Data":"13103cb9f5022641f2eed539e72b6faefd0b7e78f85cf5b453226dca7dddd1c3"}
Sep 17 23:06:32.269512 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.269468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-498dh" event={"ID":"217ae11d-95f6-4818-831d-700279f0fd51","Type":"ContainerStarted","Data":"4b40c89fa64f4c0c9c586850a8e1f86907cf75800aa381516cb9e15c425d4ae8"}
Sep 17 23:06:32.281801 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.281764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"911b0f78153f44b2f6303a5e127feea0","Type":"ContainerStarted","Data":"95a8063a6d27492bf132bbd5e6f550276fcfe896f44d8917995a16dcdc9f05da"}
Sep 17 23:06:32.295977 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.293056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hv4vh" event={"ID":"8896e89b-3c26-4006-95f0-5620d676cb48","Type":"ContainerStarted","Data":"4f42da47bf0c496588eaa0e0a7c5ae592362e4f57c62c8252a63e88252da0ae4"}
Sep 17 23:06:32.600416 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.600328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:32.600585 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.600529    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:32.600585 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.600548    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:32.600585 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.600561    2580 projected.go:196] Error preparing data for projected volume kube-api-access-j6dnd for pod openshift-network-diagnostics/network-check-target-xrmrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:32.600683 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.600650    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd podName:8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e nodeName:}" failed. No retries permitted until 2026-09-17 23:06:34.60063152 +0000 UTC m=+5.030491323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6dnd" (UniqueName: "kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd") pod "network-check-target-xrmrc" (UID: "8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:32.700737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.700703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:32.700962 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:32.700758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:32.700962 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.700923    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:32.701085 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.700986    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret podName:26053f12-63eb-48b5-a610-95ce5cd2c941 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:34.700967026 +0000 UTC m=+5.130826829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret") pod "global-pull-secret-syncer-zlv7c" (UID: "26053f12-63eb-48b5-a610-95ce5cd2c941") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:32.701382 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.701362    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:32.701452 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:32.701419    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs podName:f6a51291-55ed-41ea-8029-bf5b677956a8 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:34.701402545 +0000 UTC m=+5.131262364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs") pod "network-metrics-daemon-njwnq" (UID: "f6a51291-55ed-41ea-8029-bf5b677956a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:33.013029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:33.011246    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:31 +0000 UTC" deadline="2028-06-21 22:05:50.895442142 +0000 UTC"
Sep 17 23:06:33.013029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:33.011300    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15430h59m17.88414515s"
Sep 17 23:06:33.103058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:33.102362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:33.103058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:33.102427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:33.103058 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:33.102562    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:33.103058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:33.102608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:33.103058 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:33.102681    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:33.103058 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:33.102854    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:33.504462 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:33.504417    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:34.624001 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:34.623236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:34.624001 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.623466    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:34.624001 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.623491    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:34.624001 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.623505    2580 projected.go:196] Error preparing data for projected volume kube-api-access-j6dnd for pod openshift-network-diagnostics/network-check-target-xrmrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:34.624001 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.623566    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd podName:8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e nodeName:}" failed. No retries permitted until 2026-09-17 23:06:38.623546663 +0000 UTC m=+9.053406465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6dnd" (UniqueName: "kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd") pod "network-check-target-xrmrc" (UID: "8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:34.724815 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:34.723968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:34.724815 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:34.724076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:34.724815 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.724229    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:34.724815 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.724298    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs podName:f6a51291-55ed-41ea-8029-bf5b677956a8 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:38.724276759 +0000 UTC m=+9.154136561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs") pod "network-metrics-daemon-njwnq" (UID: "f6a51291-55ed-41ea-8029-bf5b677956a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:34.724815 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.724704    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:34.724815 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:34.724759    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret podName:26053f12-63eb-48b5-a610-95ce5cd2c941 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:38.724742236 +0000 UTC m=+9.154602035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret") pod "global-pull-secret-syncer-zlv7c" (UID: "26053f12-63eb-48b5-a610-95ce5cd2c941") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:35.102514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:35.102480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:35.102701 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:35.102479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:35.102701 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:35.102672    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:35.102831 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:35.102743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:35.102831 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:35.102492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:35.102979 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:35.102862    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:37.102648 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:37.102600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:37.103151 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:37.102743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:37.103216 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:37.103204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:37.103311 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:37.103289    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:37.103410 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:37.103360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:37.103527 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:37.103454    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:38.659001 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:38.658957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:38.659466 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.659141    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:38.659466 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.659170    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:38.659466 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.659186    2580 projected.go:196] Error preparing data for projected volume kube-api-access-j6dnd for pod openshift-network-diagnostics/network-check-target-xrmrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:38.659466 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.659261    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd podName:8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.659238742 +0000 UTC m=+17.089098544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6dnd" (UniqueName: "kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd") pod "network-check-target-xrmrc" (UID: "8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:38.760074 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:38.760031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:38.760272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:38.760154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:38.760272 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.760185    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:38.760272 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.760268    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs podName:f6a51291-55ed-41ea-8029-bf5b677956a8 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.760244007 +0000 UTC m=+17.190103805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs") pod "network-metrics-daemon-njwnq" (UID: "f6a51291-55ed-41ea-8029-bf5b677956a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:38.760470 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.760300    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:38.760470 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:38.760397    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret podName:26053f12-63eb-48b5-a610-95ce5cd2c941 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.760341758 +0000 UTC m=+17.190201575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret") pod "global-pull-secret-syncer-zlv7c" (UID: "26053f12-63eb-48b5-a610-95ce5cd2c941") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:39.101936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:39.101838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:39.102118 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:39.102007    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:39.102470 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:39.102449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:39.102576 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:39.102555    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:39.102634 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:39.102615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:39.102705 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:39.102689    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:41.102310 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:41.102210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:41.102887 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:41.102342    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:41.102887 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:41.102420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:41.102887 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:41.102480    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:41.102887 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:41.102527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:41.102887 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:41.102569    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:43.101834 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:43.101785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:43.101834 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:43.101812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:43.101834 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:43.101832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:43.102303 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:43.101916    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:43.102303 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:43.102047    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:43.102303 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:43.102136    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:45.102364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:45.102320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:45.102364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:45.102345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:45.102364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:45.102358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:45.102882 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:45.102460    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:45.102882 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:45.102595    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:45.102882 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:45.102676    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:46.721182 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:46.721142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:46.721690 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.721344    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:46.721690 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.721375    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:46.721690 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.721388    2580 projected.go:196] Error preparing data for projected volume kube-api-access-j6dnd for pod openshift-network-diagnostics/network-check-target-xrmrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:46.721690 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.721463    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd podName:8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e nodeName:}" failed. No retries permitted until 2026-09-17 23:07:02.721443213 +0000 UTC m=+33.151303029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6dnd" (UniqueName: "kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd") pod "network-check-target-xrmrc" (UID: "8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:46.821572 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:46.821532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:46.821749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:46.821589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:46.821749 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.821692    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:46.821892 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.821756    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs podName:f6a51291-55ed-41ea-8029-bf5b677956a8 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:02.821741205 +0000 UTC m=+33.251601000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs") pod "network-metrics-daemon-njwnq" (UID: "f6a51291-55ed-41ea-8029-bf5b677956a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:46.821892 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.821699    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:46.821892 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:46.821861    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret podName:26053f12-63eb-48b5-a610-95ce5cd2c941 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:02.821842699 +0000 UTC m=+33.251702513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret") pod "global-pull-secret-syncer-zlv7c" (UID: "26053f12-63eb-48b5-a610-95ce5cd2c941") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:47.102206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:47.102121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:47.102206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:47.102172    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:47.102422 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:47.102247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:47.102422 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:47.102264    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:47.102422 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:47.102393    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:47.102538 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:47.102482    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:49.102191 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:49.102146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:49.102749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:49.102146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:49.102749 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:49.102278    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:49.102749 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:49.102380    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:49.102749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:49.102148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:49.102749 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:49.102530    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:51.101818 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:51.101773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:51.102282 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:51.101794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:51.102282 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:51.101900    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:51.102282 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:51.101925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:51.102282 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:51.102005    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:51.102282 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:51.102077    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:52.345750 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.345359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk" event={"ID":"170daf82-cc47-44fb-a6d9-141059340583","Type":"ContainerStarted","Data":"b2165f05095ff05f387b682929c6252be25a240803a102295df9534225b78cb0"}
Sep 17 23:06:52.347415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.347387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sk49c" event={"ID":"4cf297b0-5433-4350-879a-8447d2e1d39c","Type":"ContainerStarted","Data":"4858713a02e8494fb3eb11bbae79f9606e487ae6c54fec6513aa20977043ba37"}
Sep 17 23:06:52.352812 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.352772    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:06:52.353235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.353203    2580 generic.go:356] "Generic (PLEG): container finished" podID="d97c14bc-f869-4487-b479-4576d1dd4493" containerID="7cd6b2b4cb661ebcd728d0576c268b96d0445b91287d51841f4f701ae8906499" exitCode=1
Sep 17 23:06:52.353324 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.353300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"9a92fb5b59fa7160173d53e20e0e2b4b236f0c3c863813608c90c2e563711472"}
Sep 17 23:06:52.353389 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.353370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"f0198d89bd4569fc83ee046f3bc7ce9f3511c09485abca84f5f410d8c3b80ade"}
Sep 17 23:06:52.353445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.353428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"4cfecc72d0be3afade47c579c3ade47b6682e9c4e69055b96753c93094ecbb9c"}
Sep 17 23:06:52.353706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.353477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"09b7c273c0ecfc437fdccecb2665f46f087732905a9d61dfe477f22a8f7277dc"}
Sep 17 23:06:52.353706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.353509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerDied","Data":"7cd6b2b4cb661ebcd728d0576c268b96d0445b91287d51841f4f701ae8906499"}
Sep 17 23:06:52.353706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.353548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"d56b704ae40e58f1d7c3b1ee9db2d82e08ad3188f6d10443424c205514fff806"}
Sep 17 23:06:52.358001 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.357532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" event={"ID":"21013831982d43c783952b1c4eba40a7","Type":"ContainerStarted","Data":"d636d6c44d701fd0a448e2523667854c6adad940a86fe99761c31211340cca8a"}
Sep 17 23:06:52.363581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.363548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vvrj6" event={"ID":"5ed5a1fa-247e-4efb-b57f-39fd524d7981","Type":"ContainerStarted","Data":"f7beaff178ad77f328935de9b50cc16fa9e7bf1acf8d74b0c25cd48fcd879c85"}
Sep 17 23:06:52.365931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.365543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-498dh" event={"ID":"217ae11d-95f6-4818-831d-700279f0fd51","Type":"ContainerStarted","Data":"9e90738532824edf1aedffdc35d46b60f9924ef9800ab6fdb6aef1042b018d1a"}
Sep 17 23:06:52.368470 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.368428    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hv4vh" event={"ID":"8896e89b-3c26-4006-95f0-5620d676cb48","Type":"ContainerStarted","Data":"35123f5832dd7f237bcfdaa56772da2ba1f371e3048c1948b1ed8e37df569839"}
Sep 17 23:06:52.384057 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.383992    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sk49c" podStartSLOduration=2.11945545 podStartE2EDuration="22.383971435s" totalImagesPullingTime="20.264515985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:52.365075952 +0000 UTC m=+22.794935784" watchObservedRunningTime="2026-09-17 23:06:52.383971435 +0000 UTC m=+22.813831275"
Sep 17 23:06:52.384738 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.384598    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6rzcj" podStartSLOduration=2.198083505 podStartE2EDuration="22.384587404s" totalImagesPullingTime="20.186503899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:52.383747269 +0000 UTC m=+22.813607087" watchObservedRunningTime="2026-09-17 23:06:52.384587404 +0000 UTC m=+22.814447223"
Sep 17 23:06:52.397708 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.397664    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" podStartSLOduration=21.397650215 podStartE2EDuration="21.397650215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:52.397311099 +0000 UTC m=+22.827170916" watchObservedRunningTime="2026-09-17 23:06:52.397650215 +0000 UTC m=+22.827510032"
Sep 17 23:06:52.414608 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.414564    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vvrj6" podStartSLOduration=1.7492928170000002 podStartE2EDuration="22.41454966s" totalImagesPullingTime="20.665256843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:52.414124879 +0000 UTC m=+22.843984699" watchObservedRunningTime="2026-09-17 23:06:52.41454966 +0000 UTC m=+22.844409478"
Sep 17 23:06:52.450185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.450141    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-498dh" podStartSLOduration=2.150864748 podStartE2EDuration="22.450123035s" totalImagesPullingTime="20.299258287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:52.450031942 +0000 UTC m=+22.879891762" watchObservedRunningTime="2026-09-17 23:06:52.450123035 +0000 UTC m=+22.879982855"
Sep 17 23:06:52.467413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.467375    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hv4vh" podStartSLOduration=2.255790102 podStartE2EDuration="22.46735695s" totalImagesPullingTime="20.211566848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:52.467100034 +0000 UTC m=+22.896959852" watchObservedRunningTime="2026-09-17 23:06:52.46735695 +0000 UTC m=+22.897216769"
Sep 17 23:06:52.560227 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.560193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:52.638177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:52.638151    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:53.102333 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.102304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:53.102616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.102423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:53.102616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.102449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:53.102616 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:53.102439    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:53.102616 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:53.102511    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:53.102828 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:53.102574    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:53.371648 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.371564    2580 generic.go:356] "Generic (PLEG): container finished" podID="911b0f78153f44b2f6303a5e127feea0" containerID="2007166af3dfdae06fa01c40f282c3e429cbe66ca88601a3ec4f72ec83bca5b5" exitCode=0
Sep 17 23:06:53.371648 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.371607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"911b0f78153f44b2f6303a5e127feea0","Type":"ContainerDied","Data":"2007166af3dfdae06fa01c40f282c3e429cbe66ca88601a3ec4f72ec83bca5b5"}
Sep 17 23:06:53.373207 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.373184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6rzcj" event={"ID":"f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62","Type":"ContainerStarted","Data":"b5dd04e0028b525bb9bf2b8cc1be5aab5733b36830cb08a80795f18869e3fb85"}
Sep 17 23:06:53.374622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.374593    2580 generic.go:356] "Generic (PLEG): container finished" podID="cefaeeee-fef2-4a38-a4a1-d26e2b452228" containerID="6622a3eb8ebd6c5ed42154d819a5cb36f301d6bbe5944ece059aa04327ff450e" exitCode=0
Sep 17 23:06:53.374734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.374662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerDied","Data":"6622a3eb8ebd6c5ed42154d819a5cb36f301d6bbe5944ece059aa04327ff450e"}
Sep 17 23:06:53.376454 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.376416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mzww5" event={"ID":"688a3748-ca64-4527-a885-37b44ede19ec","Type":"ContainerStarted","Data":"d710710e4a7faa05f8da315d3606fb5aec7f062888dfa5a875be3a4066747751"}
Sep 17 23:06:53.418434 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.418374    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mzww5" podStartSLOduration=3.14096803 podStartE2EDuration="23.418354259s" totalImagesPullingTime="20.277386229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:53.417874331 +0000 UTC m=+23.847734153" watchObservedRunningTime="2026-09-17 23:06:53.418354259 +0000 UTC m=+23.848214097"
Sep 17 23:06:53.525068 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.525045    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:06:53.539527 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.539365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:53.637271 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:53.637187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:54.050940 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.050757    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:06:53.525063144Z","UUID":"714c5295-2044-48c1-9a6e-302eb0414540","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:06:54.052573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.052544    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:06:54.052573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.052577    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:06:54.246543 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.246506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:54.306830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.306728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:54.381107 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.381067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk" event={"ID":"170daf82-cc47-44fb-a6d9-141059340583","Type":"ContainerStarted","Data":"3b38f17df772350e69130718c342109cb60324e4fc0f9f841fcab0adf160ad88"}
Sep 17 23:06:54.383185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.383160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"911b0f78153f44b2f6303a5e127feea0","Type":"ContainerStarted","Data":"8631cd96f28fc90005256b628cf4ae03e67bbd9990f153a28dfc3b68071ae449"}
Sep 17 23:06:54.400513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:54.400467    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" podStartSLOduration=23.400453281 podStartE2EDuration="23.400453281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:54.399977188 +0000 UTC m=+24.829837007" watchObservedRunningTime="2026-09-17 23:06:54.400453281 +0000 UTC m=+24.830313120"
Sep 17 23:06:55.102473 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.102441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:55.102734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.102441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:55.102734 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:55.102559    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:55.102734 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:55.102668    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:55.102734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.102445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:55.102978 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:55.102828    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:55.133769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.133733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:55.235633 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.235607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:55.339970 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.339935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:55.388357 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.388327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk" event={"ID":"170daf82-cc47-44fb-a6d9-141059340583","Type":"ContainerStarted","Data":"bf459f064c971e2400bf04eaae94249d20c8360b60c09dd53bca0c8d6f6110e7"}
Sep 17 23:06:55.393713 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.393687    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:06:55.394190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.394167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"985bad3403a5dece530f118b2568734dcda90eae645c485fa05a84fc37ad3f8e"}
Sep 17 23:06:55.408230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.408183    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mbzhk" podStartSLOduration=3.249573106 podStartE2EDuration="25.408170078s" totalImagesPullingTime="22.158596972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:55.407823378 +0000 UTC m=+25.837683196" watchObservedRunningTime="2026-09-17 23:06:55.408170078 +0000 UTC m=+25.838029896"
Sep 17 23:06:55.423000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.422966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:55.423599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.423583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:55.568441 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.568402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:55.672567 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.672479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:55.769394 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.769364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:55.869728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.869687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:55.971059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:55.970975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:56.409979 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:56.409947    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6rzcj_f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62/dns-node-resolver/0.log"
Sep 17 23:06:57.101893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.101856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:57.101893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.101875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:57.102092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.101855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:57.102092 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:57.101962    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:57.102092 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:57.102054    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:57.102200 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:57.102116    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:57.403477 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.403277    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:06:57.404128 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.403653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"4efd4c8311c1c27d062841b5e49a8901b5e4119583cf425468added54439dd4e"}
Sep 17 23:06:57.404128 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.404013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:57.404586 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.404152    2580 scope.go:117] "RemoveContainer" containerID="7cd6b2b4cb661ebcd728d0576c268b96d0445b91287d51841f4f701ae8906499"
Sep 17 23:06:57.421570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.421185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:57.517753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.517539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:57.610250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.610229    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hv4vh_8896e89b-3c26-4006-95f0-5620d676cb48/node-ca/0.log"
Sep 17 23:06:57.617999 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:57.617929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:58.407368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.407339    2580 generic.go:356] "Generic (PLEG): container finished" podID="cefaeeee-fef2-4a38-a4a1-d26e2b452228" containerID="d7a76b0366d02159c3dc3a3bc9e44f2ac61e73540bef46f7e6aeef29bdd8374b" exitCode=0
Sep 17 23:06:58.407681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.407421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerDied","Data":"d7a76b0366d02159c3dc3a3bc9e44f2ac61e73540bef46f7e6aeef29bdd8374b"}
Sep 17 23:06:58.413542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.413522    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:06:58.413923 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.413870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" event={"ID":"d97c14bc-f869-4487-b479-4576d1dd4493","Type":"ContainerStarted","Data":"db08157327a35868924382ea803d71d444ca173d55ae465735b08a2143c1362e"}
Sep 17 23:06:58.414165 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.414148    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:58.414318 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.414215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:58.429915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.429890    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:06:58.443615 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.443585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:58.463421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.463379    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4" podStartSLOduration=8.140750267 podStartE2EDuration="28.463366747s" totalImagesPullingTime="20.32261648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:58.463098975 +0000 UTC m=+28.892958793" watchObservedRunningTime="2026-09-17 23:06:58.463366747 +0000 UTC m=+28.893226598"
Sep 17 23:06:58.651577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.650976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:58.651577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.651112    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 23:06:58.655531 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:58.655331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-498dh"
Sep 17 23:06:59.102429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.102345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:59.102429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.102399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:59.102632 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.102356    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:59.102632 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:59.102486    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:59.102632 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:59.102565    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:59.102774 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:59.102670    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:59.191182 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.191153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:59.292376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.292342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:59.369712 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.369681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-njwnq"]
Sep 17 23:06:59.372344 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.372323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlv7c"]
Sep 17 23:06:59.373755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.373733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:59.388582 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.388425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xrmrc"]
Sep 17 23:06:59.419941 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.419759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:06:59.419941 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.419785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:06:59.420113 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:59.419958    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:06:59.420113 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.420075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:06:59.420113 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:59.420087    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:06:59.420284 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:06:59.420219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:06:59.509089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.509065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:59.611024 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.610942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:06:59.927453 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:06:59.927424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:00.100868 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:00.100831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:00.424194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:00.424159    2580 generic.go:356] "Generic (PLEG): container finished" podID="cefaeeee-fef2-4a38-a4a1-d26e2b452228" containerID="00fca97d8eb1d19b5b92abcf8392aab992ec0c449b00672dcad5728f086da464" exitCode=0
Sep 17 23:07:00.424357 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:00.424243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerDied","Data":"00fca97d8eb1d19b5b92abcf8392aab992ec0c449b00672dcad5728f086da464"}
Sep 17 23:07:00.500304 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:00.500275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:00.751251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:00.751219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:01.102728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.102650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:01.102728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.102693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:01.102728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.102693    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:01.102974 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:01.102759    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:01.102974 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:01.102910    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:01.103041 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:01.102988    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:01.300328 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.300301    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:01.429552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.429519    2580 generic.go:356] "Generic (PLEG): container finished" podID="cefaeeee-fef2-4a38-a4a1-d26e2b452228" containerID="e13c7abb0484fbd73dc1d600347591e4f4429249d917c0acd99716076c1b05bd" exitCode=0
Sep 17 23:07:01.429669 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.429587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerDied","Data":"e13c7abb0484fbd73dc1d600347591e4f4429249d917c0acd99716076c1b05bd"}
Sep 17 23:07:01.497393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.497362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:01.899508 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:01.899482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:02.091355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:02.091323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:02.707772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:02.707734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:02.743912 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:02.743874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:02.744111 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.744085    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:07:02.744111 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.744116    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:07:02.744261 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.744129    2580 projected.go:196] Error preparing data for projected volume kube-api-access-j6dnd for pod openshift-network-diagnostics/network-check-target-xrmrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:07:02.744261 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.744196    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd podName:8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e nodeName:}" failed. No retries permitted until 2026-09-17 23:07:34.744176425 +0000 UTC m=+65.174036236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6dnd" (UniqueName: "kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd") pod "network-check-target-xrmrc" (UID: "8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:07:02.844472 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:02.844429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:02.844665 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:02.844532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:02.844665 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.844627    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:07:02.844665 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.844626    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:07:02.844773 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.844692    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs podName:f6a51291-55ed-41ea-8029-bf5b677956a8 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:34.844674288 +0000 UTC m=+65.274534104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs") pod "network-metrics-daemon-njwnq" (UID: "f6a51291-55ed-41ea-8029-bf5b677956a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:07:02.844773 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:02.844712    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret podName:26053f12-63eb-48b5-a610-95ce5cd2c941 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:34.844702402 +0000 UTC m=+65.274562203 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret") pod "global-pull-secret-syncer-zlv7c" (UID: "26053f12-63eb-48b5-a610-95ce5cd2c941") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:07:02.890202 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:02.890162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:03.082138 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.082056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:03.102219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.102185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:03.102379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.102191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:03.102379 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:03.102310    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:03.102511 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:03.102390    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:03.102511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.102191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:03.102511 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:03.102461    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:03.182692 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.182649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:03.492637 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.492600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:03.701238 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.701202    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:03.809752 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:03.809664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:04.288842 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:04.288791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:04.524105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:04.524063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:04.938426 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:04.938330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:05.102456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:05.102423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:05.102628 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:05.102423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:05.102628 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:05.102562    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:05.102628 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:05.102423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:05.102628 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:05.102618    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:05.102825 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:05.102676    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:05.123552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:05.123521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:05.399881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:05.399854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:05.498607 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:05.498578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:05.694414 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:05.694342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:06.098030 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:06.097953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:06.445323 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:06.445294    2580 generic.go:356] "Generic (PLEG): container finished" podID="cefaeeee-fef2-4a38-a4a1-d26e2b452228" containerID="8c21773584178292661e70d581622c76d1f348d66295a0337e8edf8a19d751bb" exitCode=0
Sep 17 23:07:06.445662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:06.445347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerDied","Data":"8c21773584178292661e70d581622c76d1f348d66295a0337e8edf8a19d751bb"}
Sep 17 23:07:06.888192 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:06.888156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.102021 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.101979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:07.102021 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.102021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:07.102264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.101982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:07.102264 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:07.102112    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:07.102264 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:07.102192    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:07.102264 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:07.102231    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:07.284901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.284792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.451161 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.451132    2580 generic.go:356] "Generic (PLEG): container finished" podID="cefaeeee-fef2-4a38-a4a1-d26e2b452228" containerID="f910cc8526cc4865938d7ea385ca21fd1dc00388a488d4b3bac88aa719a5f0ca" exitCode=0
Sep 17 23:07:07.451572 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.451201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerDied","Data":"f910cc8526cc4865938d7ea385ca21fd1dc00388a488d4b3bac88aa719a5f0ca"}
Sep 17 23:07:07.493473 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.493440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.693363 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:07.693333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:08.087346 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:08.087310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:08.409249 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:08.409220    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:08.456830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:08.456778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" event={"ID":"cefaeeee-fef2-4a38-a4a1-d26e2b452228","Type":"ContainerStarted","Data":"e8e7bffe899cdc3debacbaf4925b55c435899acd3bf20b67c4dc93977f029c88"}
Sep 17 23:07:08.481305 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:08.481248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rzs4g" podStartSLOduration=8.461133667 podStartE2EDuration="38.481232576s" totalImagesPullingTime="30.020098909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:08.480772764 +0000 UTC m=+38.910632583" watchObservedRunningTime="2026-09-17 23:07:08.481232576 +0000 UTC m=+38.911092391"
Sep 17 23:07:08.488484 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:08.488460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.093011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.092976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.102104 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.102067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:09.102306 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.102076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:09.102306 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.102198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:09.102306 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:09.102194    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:09.102306 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:09.102279    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:09.102456 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:09.102338    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:09.187076 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.186991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.291465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.291432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.649526 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.649496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.736429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.736397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.889020 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:09.888989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:11.102341 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:11.102310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:11.102829 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:11.102310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:11.102829 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:11.102412    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:11.102829 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:11.102310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:11.102829 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:11.102493    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:11.102829 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:11.102593    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:12.092759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:12.092726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:12.192180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:12.192140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:13.101816 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:13.101759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:13.102001 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:13.101869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:13.102001 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:13.101962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:13.102001 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:13.101977    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:13.102126 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:13.102038    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:13.102126 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:13.102092    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:13.655413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:13.655381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:13.756981 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:13.756946    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:14.157132 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:14.157102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:14.334848 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:14.334810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:14.393789 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:14.393754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:14.529239 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:14.529165    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:14.629862 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:14.629830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:15.101845 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.101817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:15.101845 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.101851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:15.102345 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.101906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:15.102345 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:15.101922    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:15.102345 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:15.101970    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:15.102345 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:15.102052    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:15.143503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.143472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:15.351072 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.351041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:15.542287 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.542260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:15.609339 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.609310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:15.715521 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:15.715492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:17.101970 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:17.101887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:17.102297 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:17.101887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:17.102297 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:17.102090    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:17.102297 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:17.101989    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:17.102297 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:17.101895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:17.102297 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:17.102173    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:19.102424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:19.102385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:19.102865 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:19.102403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:19.102865 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:19.102508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:19.102865 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:19.102509    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:19.102865 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:19.102568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:19.102865 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:19.102664    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:21.101900 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:21.101866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:21.101900 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:21.101903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:21.102317 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:21.101904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:21.102317 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:21.101969    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zlv7c" podUID="26053f12-63eb-48b5-a610-95ce5cd2c941"
Sep 17 23:07:21.102317 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:21.102047    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njwnq" podUID="f6a51291-55ed-41ea-8029-bf5b677956a8"
Sep 17 23:07:21.102317 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:21.102107    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xrmrc" podUID="8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e"
Sep 17 23:07:21.422974 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:21.422943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:21.661032 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:21.661004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:21.787200 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:21.787122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:21.987605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:21.987573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.087564 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.087496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.241585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.241555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.472702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.472673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.566391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.566334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.819742 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.819674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.823392 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.823372    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeReady"
Sep 17 23:07:22.823523 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.823513    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:07:22.896517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.896484    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bt4qj"]
Sep 17 23:07:22.908988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.908963    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xp5xd"]
Sep 17 23:07:22.909096 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.909075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:22.911974 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.911951    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:07:22.911974 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.911962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:07:22.912223 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.911951    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:07:22.914792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.914761    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ljrmd\""
Sep 17 23:07:22.919018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.918996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bt4qj"]
Sep 17 23:07:22.919123 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.919027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xp5xd"]
Sep 17 23:07:22.919123 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.919115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:22.929175 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.929154    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbrzm\""
Sep 17 23:07:22.929277 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.929189    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:07:22.929277 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.929211    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:07:22.929392 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.929376    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:07:22.929932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.929920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:07:22.930733 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.930715    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sz9q8"]
Sep 17 23:07:22.947368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.947340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.947368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.947365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:22.947588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.947431    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sz9q8"]
Sep 17 23:07:22.950644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.950623    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:07:22.950814 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.950786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:07:22.950894 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.950818    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-62qw9\""
Sep 17 23:07:22.990638 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c45bc469-07af-45ce-bfb0-aa730901da10-cert\") pod \"ingress-canary-bt4qj\" (UID: \"c45bc469-07af-45ce-bfb0-aa730901da10\") " pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:22.990638 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/664bd601-ed30-4dab-9ba5-131d4b86e869-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/664bd601-ed30-4dab-9ba5-131d4b86e869-data-volume\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d58559-fe6c-4175-9e0d-664079115ee3-tmp-dir\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/664bd601-ed30-4dab-9ba5-131d4b86e869-crio-socket\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrpmr\" (UniqueName: \"kubernetes.io/projected/c45bc469-07af-45ce-bfb0-aa730901da10-kube-api-access-lrpmr\") pod \"ingress-canary-bt4qj\" (UID: \"c45bc469-07af-45ce-bfb0-aa730901da10\") " pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4pjb\" (UniqueName: \"kubernetes.io/projected/664bd601-ed30-4dab-9ba5-131d4b86e869-kube-api-access-g4pjb\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d58559-fe6c-4175-9e0d-664079115ee3-config-volume\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdrs\" (UniqueName: \"kubernetes.io/projected/b7d58559-fe6c-4175-9e0d-664079115ee3-kube-api-access-stdrs\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7d58559-fe6c-4175-9e0d-664079115ee3-metrics-tls\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:22.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:22.990932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/664bd601-ed30-4dab-9ba5-131d4b86e869-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.035982 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.035941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:23.091483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7d58559-fe6c-4175-9e0d-664079115ee3-metrics-tls\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.091483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/664bd601-ed30-4dab-9ba5-131d4b86e869-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.091694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c45bc469-07af-45ce-bfb0-aa730901da10-cert\") pod \"ingress-canary-bt4qj\" (UID: \"c45bc469-07af-45ce-bfb0-aa730901da10\") " pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:23.091694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/664bd601-ed30-4dab-9ba5-131d4b86e869-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.091694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/664bd601-ed30-4dab-9ba5-131d4b86e869-data-volume\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.091694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d58559-fe6c-4175-9e0d-664079115ee3-tmp-dir\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.091694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/664bd601-ed30-4dab-9ba5-131d4b86e869-crio-socket\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.091694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpmr\" (UniqueName: \"kubernetes.io/projected/c45bc469-07af-45ce-bfb0-aa730901da10-kube-api-access-lrpmr\") pod \"ingress-canary-bt4qj\" (UID: \"c45bc469-07af-45ce-bfb0-aa730901da10\") " pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:23.091694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4pjb\" (UniqueName: \"kubernetes.io/projected/664bd601-ed30-4dab-9ba5-131d4b86e869-kube-api-access-g4pjb\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.092055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d58559-fe6c-4175-9e0d-664079115ee3-config-volume\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.092055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdrs\" (UniqueName: \"kubernetes.io/projected/b7d58559-fe6c-4175-9e0d-664079115ee3-kube-api-access-stdrs\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.092055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.091965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/664bd601-ed30-4dab-9ba5-131d4b86e869-data-volume\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.092055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.092001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7d58559-fe6c-4175-9e0d-664079115ee3-tmp-dir\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.092245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.092145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/664bd601-ed30-4dab-9ba5-131d4b86e869-crio-socket\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.092482 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.092460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/664bd601-ed30-4dab-9ba5-131d4b86e869-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.092540 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.092460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d58559-fe6c-4175-9e0d-664079115ee3-config-volume\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.095575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.095553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b7d58559-fe6c-4175-9e0d-664079115ee3-metrics-tls\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.095677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.095650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/664bd601-ed30-4dab-9ba5-131d4b86e869-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.095731 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.095716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c45bc469-07af-45ce-bfb0-aa730901da10-cert\") pod \"ingress-canary-bt4qj\" (UID: \"c45bc469-07af-45ce-bfb0-aa730901da10\") " pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:23.100215 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.100193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdrs\" (UniqueName: \"kubernetes.io/projected/b7d58559-fe6c-4175-9e0d-664079115ee3-kube-api-access-stdrs\") pod \"dns-default-sz9q8\" (UID: \"b7d58559-fe6c-4175-9e0d-664079115ee3\") " pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.100307 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.100235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4pjb\" (UniqueName: \"kubernetes.io/projected/664bd601-ed30-4dab-9ba5-131d4b86e869-kube-api-access-g4pjb\") pod \"insights-runtime-extractor-xp5xd\" (UID: \"664bd601-ed30-4dab-9ba5-131d4b86e869\") " pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.100659 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.100637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrpmr\" (UniqueName: \"kubernetes.io/projected/c45bc469-07af-45ce-bfb0-aa730901da10-kube-api-access-lrpmr\") pod \"ingress-canary-bt4qj\" (UID: \"c45bc469-07af-45ce-bfb0-aa730901da10\") " pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:23.102458 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.102437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:23.102589 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.102526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:23.102589 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.102552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:23.105323 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.105301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:07:23.105578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.105560    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p8cnd\""
Sep 17 23:07:23.105671 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.105565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fkqjx\""
Sep 17 23:07:23.105671 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.105594    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:07:23.105671 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.105635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:07:23.105901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.105889    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:07:23.136042 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.136009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:23.219907 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.219870    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bt4qj"
Sep 17 23:07:23.234045 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.234017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xp5xd"
Sep 17 23:07:23.256547 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.256523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:23.560552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.560526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bt4qj"]
Sep 17 23:07:23.561381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.561362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xp5xd"]
Sep 17 23:07:23.563023 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:23.562996    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664bd601_ed30_4dab_9ba5_131d4b86e869.slice/crio-5ad4c755e654384cfe60e6a1b254d2273068fec284cdefeefac7dc2599605edd WatchSource:0}: Error finding container 5ad4c755e654384cfe60e6a1b254d2273068fec284cdefeefac7dc2599605edd: Status 404 returned error can't find the container with id 5ad4c755e654384cfe60e6a1b254d2273068fec284cdefeefac7dc2599605edd
Sep 17 23:07:23.563390 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:23.563320    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45bc469_07af_45ce_bfb0_aa730901da10.slice/crio-cb30a91dc3d941835abe6d6fd8d46b5cc2f6bb83e79a60835a6a1f16c201f45d WatchSource:0}: Error finding container cb30a91dc3d941835abe6d6fd8d46b5cc2f6bb83e79a60835a6a1f16c201f45d: Status 404 returned error can't find the container with id cb30a91dc3d941835abe6d6fd8d46b5cc2f6bb83e79a60835a6a1f16c201f45d
Sep 17 23:07:23.565399 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.565375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sz9q8"]
Sep 17 23:07:23.569396 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:23.569378    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d58559_fe6c_4175_9e0d_664079115ee3.slice/crio-2d972c87840e926ad75df3b1fa96410b924ff25fed9c59de48ec75dda7dc3405 WatchSource:0}: Error finding container 2d972c87840e926ad75df3b1fa96410b924ff25fed9c59de48ec75dda7dc3405: Status 404 returned error can't find the container with id 2d972c87840e926ad75df3b1fa96410b924ff25fed9c59de48ec75dda7dc3405
Sep 17 23:07:23.576915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.576892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:23.678922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:23.678898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:24.495890 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:24.495852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bt4qj" event={"ID":"c45bc469-07af-45ce-bfb0-aa730901da10","Type":"ContainerStarted","Data":"cb30a91dc3d941835abe6d6fd8d46b5cc2f6bb83e79a60835a6a1f16c201f45d"}
Sep 17 23:07:24.497424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:24.497396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sz9q8" event={"ID":"b7d58559-fe6c-4175-9e0d-664079115ee3","Type":"ContainerStarted","Data":"2d972c87840e926ad75df3b1fa96410b924ff25fed9c59de48ec75dda7dc3405"}
Sep 17 23:07:24.499548 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:24.499506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xp5xd" event={"ID":"664bd601-ed30-4dab-9ba5-131d4b86e869","Type":"ContainerStarted","Data":"347ac0c653ced3b003e74b5e9b77f1734c8037d99cecec234da0e31e147dde04"}
Sep 17 23:07:24.499665 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:24.499569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xp5xd" event={"ID":"664bd601-ed30-4dab-9ba5-131d4b86e869","Type":"ContainerStarted","Data":"51b64ad75b78f7746ccdf357dbfb88039d9d1da70cf33ab70fabbf430ea2ca4e"}
Sep 17 23:07:24.499665 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:24.499596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xp5xd" event={"ID":"664bd601-ed30-4dab-9ba5-131d4b86e869","Type":"ContainerStarted","Data":"5ad4c755e654384cfe60e6a1b254d2273068fec284cdefeefac7dc2599605edd"}
Sep 17 23:07:27.374005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.373967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:27.438645 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.438617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:27.510011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.509979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sz9q8" event={"ID":"b7d58559-fe6c-4175-9e0d-664079115ee3","Type":"ContainerStarted","Data":"2af967666c5741b485d786c7e71e3eb5d9862ee9b06b3714b2f5638f9e6e7c7b"}
Sep 17 23:07:27.510170 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.510066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sz9q8" event={"ID":"b7d58559-fe6c-4175-9e0d-664079115ee3","Type":"ContainerStarted","Data":"eeeecb5b426b699e39e63cbe06f8c62982983229673cf15ce34ed41bf7a55b87"}
Sep 17 23:07:27.510170 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.510099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:27.512205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.512169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xp5xd" event={"ID":"664bd601-ed30-4dab-9ba5-131d4b86e869","Type":"ContainerStarted","Data":"2ce95b85b089e6bde35a4df0ee66702d0eb134f8513bf2a9e974b189369426e6"}
Sep 17 23:07:27.513700 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.513671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bt4qj" event={"ID":"c45bc469-07af-45ce-bfb0-aa730901da10","Type":"ContainerStarted","Data":"50b11457a69ec0aed72fd7a03c8eea82075a1c5b84d0260799379cd018af945c"}
Sep 17 23:07:27.528061 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.528016    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sz9q8" podStartSLOduration=2.645670632 podStartE2EDuration="5.528002594s" totalImagesPullingTime="2.882331962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:27.527386963 +0000 UTC m=+57.957246782" watchObservedRunningTime="2026-09-17 23:07:27.528002594 +0000 UTC m=+57.957862445"
Sep 17 23:07:27.542431 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.542383    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bt4qj" podStartSLOduration=2.647987413 podStartE2EDuration="5.542365948s" totalImagesPullingTime="2.894378535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:27.541773585 +0000 UTC m=+57.971633403" watchObservedRunningTime="2026-09-17 23:07:27.542365948 +0000 UTC m=+57.972225766"
Sep 17 23:07:27.559555 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:27.559512    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xp5xd" podStartSLOduration=2.8556033210000002 podStartE2EDuration="5.559498267s" totalImagesPullingTime="2.703894946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:07:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:27.558690366 +0000 UTC m=+57.988550183" watchObservedRunningTime="2026-09-17 23:07:27.559498267 +0000 UTC m=+57.989358079"
Sep 17 23:07:29.007602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:29.007566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:29.063939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:29.063908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:30.338560 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:30.338522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:30.437427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:30.437393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rfdm4"
Sep 17 23:07:30.438668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:30.438645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:30.539861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:30.539829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:30.640426 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:30.640395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:30.740855 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:30.740826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:31.173296 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.173265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:31.486027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.485942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:31.548595 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.548562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:31.959552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.959519    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58f5c889b8-rx749"]
Sep 17 23:07:31.962206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.962183    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:31.964309 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.964286    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:07:31.964430 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.964372    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:07:31.965211 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.965183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:07:31.965211 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.965196    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:07:31.965434 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.965260    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t72j2\""
Sep 17 23:07:31.965434 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.965269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:07:31.965596 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.965457    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:07:31.965596 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.965460    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:07:31.973406 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.973370    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 23:07:31.974669 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:31.974611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f5c889b8-rx749"]
Sep 17 23:07:32.058167 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.058105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-oauth-serving-cert\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.058167 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.058163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-oauth-config\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.058391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.058216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-trusted-ca-bundle\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.058391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.058254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl74z\" (UniqueName: \"kubernetes.io/projected/32c5b3f7-46a2-49a4-9464-84168ea9c74d-kube-api-access-hl74z\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.058391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.058276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-service-ca\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.058391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.058373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-serving-cert\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.058508 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.058400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-config\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.159551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.159513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-trusted-ca-bundle\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.159551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.159556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl74z\" (UniqueName: \"kubernetes.io/projected/32c5b3f7-46a2-49a4-9464-84168ea9c74d-kube-api-access-hl74z\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.159825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.159578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-service-ca\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.159825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.159632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-serving-cert\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.159825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.159647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-config\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.159825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.159668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-oauth-serving-cert\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.159825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.159691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-oauth-config\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.160487 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.160459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-service-ca\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.160598 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.160537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-trusted-ca-bundle\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.160598 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.160535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-oauth-serving-cert\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.160672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.160640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-config\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.162348 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.162326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-oauth-config\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.162461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.162439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-serving-cert\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.167194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.167175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl74z\" (UniqueName: \"kubernetes.io/projected/32c5b3f7-46a2-49a4-9464-84168ea9c74d-kube-api-access-hl74z\") pod \"console-58f5c889b8-rx749\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") " pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.272252 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.272155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:32.278739 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.278706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:32.308543 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.307709    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"]
Sep 17 23:07:32.310298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.310274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.312747 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.312715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 23:07:32.312890 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.312760    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:07:32.313089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.313060    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:07:32.313089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.313063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 23:07:32.313225 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.313165    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q6nfk\""
Sep 17 23:07:32.313292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.313273    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:07:32.322248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.322198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"]
Sep 17 23:07:32.354116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.354035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:32.360569 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.360525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/429aa7db-1cda-455d-844a-0e0a1e30b524-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.360710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.360593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/429aa7db-1cda-455d-844a-0e0a1e30b524-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.360710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.360644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/429aa7db-1cda-455d-844a-0e0a1e30b524-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.360830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.360712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xbwx\" (UniqueName: \"kubernetes.io/projected/429aa7db-1cda-455d-844a-0e0a1e30b524-kube-api-access-6xbwx\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.461103 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.461071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/429aa7db-1cda-455d-844a-0e0a1e30b524-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.461256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.461129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/429aa7db-1cda-455d-844a-0e0a1e30b524-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.461256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.461186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbwx\" (UniqueName: \"kubernetes.io/projected/429aa7db-1cda-455d-844a-0e0a1e30b524-kube-api-access-6xbwx\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.461256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.461236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/429aa7db-1cda-455d-844a-0e0a1e30b524-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.461762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.461738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/429aa7db-1cda-455d-844a-0e0a1e30b524-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.463709 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.463689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/429aa7db-1cda-455d-844a-0e0a1e30b524-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.463788 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.463748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/429aa7db-1cda-455d-844a-0e0a1e30b524-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.470344 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.470323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbwx\" (UniqueName: \"kubernetes.io/projected/429aa7db-1cda-455d-844a-0e0a1e30b524-kube-api-access-6xbwx\") pod \"prometheus-operator-7bd5d56d6-ppbfj\" (UID: \"429aa7db-1cda-455d-844a-0e0a1e30b524\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.501472 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.501443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:32.546620 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.546537    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f5c889b8-rx749"]
Sep 17 23:07:32.549454 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:32.549430    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c5b3f7_46a2_49a4_9464_84168ea9c74d.slice/crio-d2136faa6a54e476fc9f90eca080e0cd80f0c21c65c6ab0da278250dfa686fc9 WatchSource:0}: Error finding container d2136faa6a54e476fc9f90eca080e0cd80f0c21c65c6ab0da278250dfa686fc9: Status 404 returned error can't find the container with id d2136faa6a54e476fc9f90eca080e0cd80f0c21c65c6ab0da278250dfa686fc9
Sep 17 23:07:32.621262 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.621236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"
Sep 17 23:07:32.891811 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:32.891760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj"]
Sep 17 23:07:32.896387 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:32.896010    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429aa7db_1cda_455d_844a_0e0a1e30b524.slice/crio-d3b9b9ebdc4c0a5a0593883e795cd0713fd64adcf51a7b6a54eda3a75a7ab183 WatchSource:0}: Error finding container d3b9b9ebdc4c0a5a0593883e795cd0713fd64adcf51a7b6a54eda3a75a7ab183: Status 404 returned error can't find the container with id d3b9b9ebdc4c0a5a0593883e795cd0713fd64adcf51a7b6a54eda3a75a7ab183
Sep 17 23:07:33.534971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:33.534935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj" event={"ID":"429aa7db-1cda-455d-844a-0e0a1e30b524","Type":"ContainerStarted","Data":"d3b9b9ebdc4c0a5a0593883e795cd0713fd64adcf51a7b6a54eda3a75a7ab183"}
Sep 17 23:07:33.536647 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:33.536610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f5c889b8-rx749" event={"ID":"32c5b3f7-46a2-49a4-9464-84168ea9c74d","Type":"ContainerStarted","Data":"d2136faa6a54e476fc9f90eca080e0cd80f0c21c65c6ab0da278250dfa686fc9"}
Sep 17 23:07:34.542643 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.542600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj" event={"ID":"429aa7db-1cda-455d-844a-0e0a1e30b524","Type":"ContainerStarted","Data":"6aecdb781ce5ab2569d6b9d00fb6dbb2cea6fb6028cb4311f127b25cc8baa39f"}
Sep 17 23:07:34.543011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.542668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj" event={"ID":"429aa7db-1cda-455d-844a-0e0a1e30b524","Type":"ContainerStarted","Data":"ae918611e72aaf8f5c7c4b3a77b29a4fb297af9c0c6ab59435749927e6134294"}
Sep 17 23:07:34.562287 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.562225    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-ppbfj" podStartSLOduration=1.08999431 podStartE2EDuration="2.562204006s" totalImagesPullingTime="1.472209696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:34.561192959 +0000 UTC m=+64.991052781" watchObservedRunningTime="2026-09-17 23:07:34.562204006 +0000 UTC m=+64.992063823"
Sep 17 23:07:34.573714 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.573685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:34.666701 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.666658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:34.783874 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.783833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:34.786445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.786417    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:07:34.796838 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.796814    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:07:34.807920 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.807896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6dnd\" (UniqueName: \"kubernetes.io/projected/8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e-kube-api-access-j6dnd\") pod \"network-check-target-xrmrc\" (UID: \"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e\") " pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:34.823089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.823061    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fkqjx\""
Sep 17 23:07:34.831360 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.831340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:34.885104 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.885068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:34.885286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.885120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:34.887358 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.887338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:07:34.887492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.887344    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:07:34.898728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.898705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/26053f12-63eb-48b5-a610-95ce5cd2c941-original-pull-secret\") pod \"global-pull-secret-syncer-zlv7c\" (UID: \"26053f12-63eb-48b5-a610-95ce5cd2c941\") " pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:34.898857 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:34.898826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a51291-55ed-41ea-8029-bf5b677956a8-metrics-certs\") pod \"network-metrics-daemon-njwnq\" (UID: \"f6a51291-55ed-41ea-8029-bf5b677956a8\") " pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:35.115922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:35.115838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zlv7c"
Sep 17 23:07:35.128154 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:35.128126    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p8cnd\""
Sep 17 23:07:35.136432 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:35.136402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njwnq"
Sep 17 23:07:36.427566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.427530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:36.661153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.661097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:36.677347 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.677315    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"]
Sep 17 23:07:36.714140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.714009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"]
Sep 17 23:07:36.714140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.714051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.714140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.714094    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"]
Sep 17 23:07:36.716616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.716589    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:07:36.716747 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.716638    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 23:07:36.716747 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.716596    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vzkk8\""
Sep 17 23:07:36.746323 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.746286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"]
Sep 17 23:07:36.746482 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.746342    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rbtkm"]
Sep 17 23:07:36.746482 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.746411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.748938 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.748912    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 23:07:36.749089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.748984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:07:36.749089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.749040    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nhsjf\""
Sep 17 23:07:36.749089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.749041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 23:07:36.760690 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.760657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:36.777018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.776979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.779672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.779647    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:07:36.779832 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.779762    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5vsd4\""
Sep 17 23:07:36.780025 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.780011    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:07:36.780125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.780103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:07:36.797772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.797741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c383065b-e43f-4c7b-aa91-1a1845fcd488-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.797922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.797789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.797922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.797880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c383065b-e43f-4c7b-aa91-1a1845fcd488-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.798029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.797929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edd19b0b-6819-4558-92f5-089bd1b674ef-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.798029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.797959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.798029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.798018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsts\" (UniqueName: \"kubernetes.io/projected/c383065b-e43f-4c7b-aa91-1a1845fcd488-kube-api-access-lmsts\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.798135 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.798046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-api-access-csl6r\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.798135 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.798082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.798135 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.798107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edd19b0b-6819-4558-92f5-089bd1b674ef-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.798265 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.798174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c383065b-e43f-4c7b-aa91-1a1845fcd488-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.828053 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.828014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:36.899252 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-wtmp\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsts\" (UniqueName: \"kubernetes.io/projected/c383065b-e43f-4c7b-aa91-1a1845fcd488-kube-api-access-lmsts\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.899445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-textfile\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-api-access-csl6r\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.899445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxww\" (UniqueName: \"kubernetes.io/projected/8fc37975-cb27-4d09-9c60-916d3a88a503-kube-api-access-7gxww\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fc37975-cb27-4d09-9c60-916d3a88a503-metrics-client-ca\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.899588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edd19b0b-6819-4558-92f5-089bd1b674ef-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.899716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-accelerators-collector-config\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c383065b-e43f-4c7b-aa91-1a1845fcd488-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.899716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c383065b-e43f-4c7b-aa91-1a1845fcd488-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.899716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.899716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-sys\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-root\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:36.899953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c383065b-e43f-4c7b-aa91-1a1845fcd488-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.899953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edd19b0b-6819-4558-92f5-089bd1b674ef-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.899953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.900148 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.899983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/edd19b0b-6819-4558-92f5-089bd1b674ef-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.900148 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:36.900076    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 23:07:36.900148 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:36.900144    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-tls podName:edd19b0b-6819-4558-92f5-089bd1b674ef nodeName:}" failed. No retries permitted until 2026-09-17 23:07:37.400123358 +0000 UTC m=+67.829983168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-6f4ms" (UID: "edd19b0b-6819-4558-92f5-089bd1b674ef") : secret "kube-state-metrics-tls" not found
Sep 17 23:07:36.901142 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.901114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.901473 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.901179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c383065b-e43f-4c7b-aa91-1a1845fcd488-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.901473 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.901298    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edd19b0b-6819-4558-92f5-089bd1b674ef-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.903372 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.903348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.903453 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.903348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c383065b-e43f-4c7b-aa91-1a1845fcd488-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.903639 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.903620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c383065b-e43f-4c7b-aa91-1a1845fcd488-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.908686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.908660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-api-access-csl6r\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:36.908877 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.908855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsts\" (UniqueName: \"kubernetes.io/projected/c383065b-e43f-4c7b-aa91-1a1845fcd488-kube-api-access-lmsts\") pod \"openshift-state-metrics-86c5bc79d6-2bqjt\" (UID: \"c383065b-e43f-4c7b-aa91-1a1845fcd488\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:36.929536 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:36.929504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.000551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-wtmp\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-textfile\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxww\" (UniqueName: \"kubernetes.io/projected/8fc37975-cb27-4d09-9c60-916d3a88a503-kube-api-access-7gxww\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fc37975-cb27-4d09-9c60-916d3a88a503-metrics-client-ca\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-accelerators-collector-config\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-wtmp\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-sys\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-sys\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.000794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-root\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.001159 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fc37975-cb27-4d09-9c60-916d3a88a503-root\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.001159 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.000908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-textfile\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.001159 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:37.001060    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:07:37.001159 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:37.001130    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls podName:8fc37975-cb27-4d09-9c60-916d3a88a503 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:37.501107318 +0000 UTC m=+67.930967160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls") pod "node-exporter-rbtkm" (UID: "8fc37975-cb27-4d09-9c60-916d3a88a503") : secret "node-exporter-tls" not found
Sep 17 23:07:37.001465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.001446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fc37975-cb27-4d09-9c60-916d3a88a503-metrics-client-ca\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.001701 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.001492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-accelerators-collector-config\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.003419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.003396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.013295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.013263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxww\" (UniqueName: \"kubernetes.io/projected/8fc37975-cb27-4d09-9c60-916d3a88a503-kube-api-access-7gxww\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.026023 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.025993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"
Sep 17 23:07:37.033446 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.033392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.137321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.137254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.233578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.233515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.332490 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.332396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.404895 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.404863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:37.405479 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.405450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zlv7c"]
Sep 17 23:07:37.407466 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.407444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/edd19b0b-6819-4558-92f5-089bd1b674ef-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-6f4ms\" (UID: \"edd19b0b-6819-4558-92f5-089bd1b674ef\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:37.409011 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:37.408984    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26053f12_63eb_48b5_a610_95ce5cd2c941.slice/crio-c777c599fa7e435be8649c7aba880513a4aef1282b886fb4f90075dace31504f WatchSource:0}: Error finding container c777c599fa7e435be8649c7aba880513a4aef1282b886fb4f90075dace31504f: Status 404 returned error can't find the container with id c777c599fa7e435be8649c7aba880513a4aef1282b886fb4f90075dace31504f
Sep 17 23:07:37.411506 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.411474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-njwnq"]
Sep 17 23:07:37.414654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.414633    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt"]
Sep 17 23:07:37.416320 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:37.415964    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a51291_55ed_41ea_8029_bf5b677956a8.slice/crio-4385161b05a55f6231adf773f3325d75cfcdd7c78f168d7aed4e5f1d01bffde6 WatchSource:0}: Error finding container 4385161b05a55f6231adf773f3325d75cfcdd7c78f168d7aed4e5f1d01bffde6: Status 404 returned error can't find the container with id 4385161b05a55f6231adf773f3325d75cfcdd7c78f168d7aed4e5f1d01bffde6
Sep 17 23:07:37.420329 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:37.420299    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc383065b_e43f_4c7b_aa91_1a1845fcd488.slice/crio-83b0e1200faf35b024c0b22f43980a4a80d1ef11f375b235cd702a681efdd308 WatchSource:0}: Error finding container 83b0e1200faf35b024c0b22f43980a4a80d1ef11f375b235cd702a681efdd308: Status 404 returned error can't find the container with id 83b0e1200faf35b024c0b22f43980a4a80d1ef11f375b235cd702a681efdd308
Sep 17 23:07:37.426035 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.426008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xrmrc"]
Sep 17 23:07:37.430239 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:37.430211    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc02dbe_b98b_4ab3_9fe5_fcbe41986d8e.slice/crio-731561883a9d93c2cc00d7288534c1fe7ebeaf83d9fec7e2589bf2a24db2c3e1 WatchSource:0}: Error finding container 731561883a9d93c2cc00d7288534c1fe7ebeaf83d9fec7e2589bf2a24db2c3e1: Status 404 returned error can't find the container with id 731561883a9d93c2cc00d7288534c1fe7ebeaf83d9fec7e2589bf2a24db2c3e1
Sep 17 23:07:37.505550 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.505516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:37.505712 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:37.505691    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:07:37.505778 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:07:37.505768    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls podName:8fc37975-cb27-4d09-9c60-916d3a88a503 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:38.505746001 +0000 UTC m=+68.935605812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls") pod "node-exporter-rbtkm" (UID: "8fc37975-cb27-4d09-9c60-916d3a88a503") : secret "node-exporter-tls" not found
Sep 17 23:07:37.519171 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.519140    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sz9q8"
Sep 17 23:07:37.555484 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.555453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xrmrc" event={"ID":"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e","Type":"ContainerStarted","Data":"731561883a9d93c2cc00d7288534c1fe7ebeaf83d9fec7e2589bf2a24db2c3e1"}
Sep 17 23:07:37.559222 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.559183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f5c889b8-rx749" event={"ID":"32c5b3f7-46a2-49a4-9464-84168ea9c74d","Type":"ContainerStarted","Data":"6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850"}
Sep 17 23:07:37.561850 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.560527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njwnq" event={"ID":"f6a51291-55ed-41ea-8029-bf5b677956a8","Type":"ContainerStarted","Data":"4385161b05a55f6231adf773f3325d75cfcdd7c78f168d7aed4e5f1d01bffde6"}
Sep 17 23:07:37.561850 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.561741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zlv7c" event={"ID":"26053f12-63eb-48b5-a610-95ce5cd2c941","Type":"ContainerStarted","Data":"c777c599fa7e435be8649c7aba880513a4aef1282b886fb4f90075dace31504f"}
Sep 17 23:07:37.563758 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.563704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt" event={"ID":"c383065b-e43f-4c7b-aa91-1a1845fcd488","Type":"ContainerStarted","Data":"939ea1e5d40018c236dcb4e0077ff8148efd0fbbd92a73134d189bcfc839c72b"}
Sep 17 23:07:37.563758 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.563748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt" event={"ID":"c383065b-e43f-4c7b-aa91-1a1845fcd488","Type":"ContainerStarted","Data":"83b0e1200faf35b024c0b22f43980a4a80d1ef11f375b235cd702a681efdd308"}
Sep 17 23:07:37.578647 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.578600    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58f5c889b8-rx749" podStartSLOduration=1.731271275 podStartE2EDuration="6.578581331s" totalImagesPullingTime="4.847310056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:37.576824955 +0000 UTC m=+68.006684765" watchObservedRunningTime="2026-09-17 23:07:37.578581331 +0000 UTC m=+68.008441150"
Sep 17 23:07:37.657475 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.657444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"
Sep 17 23:07:37.679861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.679830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.773749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.773671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.896845 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.896770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.959281 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.959244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-6f4ms"]
Sep 17 23:07:37.961473 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:37.961441    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd19b0b_6819_4558_92f5_089bd1b674ef.slice/crio-25311e860e8b7f96b1ff2316dd660f5de898de02527a2d1b02643dd6e346e604 WatchSource:0}: Error finding container 25311e860e8b7f96b1ff2316dd660f5de898de02527a2d1b02643dd6e346e604: Status 404 returned error can't find the container with id 25311e860e8b7f96b1ff2316dd660f5de898de02527a2d1b02643dd6e346e604
Sep 17 23:07:37.996039 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:37.995993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.096596 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.096499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.187568 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.187472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.518921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.518834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:38.522838 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.522777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fc37975-cb27-4d09-9c60-916d3a88a503-node-exporter-tls\") pod \"node-exporter-rbtkm\" (UID: \"8fc37975-cb27-4d09-9c60-916d3a88a503\") " pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:38.572950 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.572843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms" event={"ID":"edd19b0b-6819-4558-92f5-089bd1b674ef","Type":"ContainerStarted","Data":"25311e860e8b7f96b1ff2316dd660f5de898de02527a2d1b02643dd6e346e604"}
Sep 17 23:07:38.576452 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.576381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt" event={"ID":"c383065b-e43f-4c7b-aa91-1a1845fcd488","Type":"ContainerStarted","Data":"06c4dc58ddff12298f3be9e02df5ec3fee84bce72434926fa716b13543475e8e"}
Sep 17 23:07:38.589513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.589476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rbtkm"
Sep 17 23:07:38.836499 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.836403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.905584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:38.905552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:39.611336 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:39.611309    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc37975_cb27_4d09_9c60_916d3a88a503.slice/crio-490d8d32500d7e12d5c8885fd3f77f647bb492b3e89f2acbf9a2c98066cacc1f WatchSource:0}: Error finding container 490d8d32500d7e12d5c8885fd3f77f647bb492b3e89f2acbf9a2c98066cacc1f: Status 404 returned error can't find the container with id 490d8d32500d7e12d5c8885fd3f77f647bb492b3e89f2acbf9a2c98066cacc1f
Sep 17 23:07:39.618833 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:39.615859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:39.697517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:39.697372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:39.986247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:39.986216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:40.086513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.086398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:40.591401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.591361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms" event={"ID":"edd19b0b-6819-4558-92f5-089bd1b674ef","Type":"ContainerStarted","Data":"7ddd4e5cae2eeab5fbf1592375f80e6a13f75b546a37c333e2a696ede2096bda"}
Sep 17 23:07:40.591561 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.591439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms" event={"ID":"edd19b0b-6819-4558-92f5-089bd1b674ef","Type":"ContainerStarted","Data":"1cfdf617fa8e6e7be2397a0166e2b26f961303010c2dd722ecfde372c87131bf"}
Sep 17 23:07:40.591561 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.591468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms" event={"ID":"edd19b0b-6819-4558-92f5-089bd1b674ef","Type":"ContainerStarted","Data":"e932116dddb1c86f5885173ef615d92ea121c3d62c118fb711b0688ae0042b51"}
Sep 17 23:07:40.595636 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.595606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt" event={"ID":"c383065b-e43f-4c7b-aa91-1a1845fcd488","Type":"ContainerStarted","Data":"4e72e5a99734732750df0dbce3730d6a78217589cb3f90771744002be016742b"}
Sep 17 23:07:40.597688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.597659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbtkm" event={"ID":"8fc37975-cb27-4d09-9c60-916d3a88a503","Type":"ContainerStarted","Data":"490d8d32500d7e12d5c8885fd3f77f647bb492b3e89f2acbf9a2c98066cacc1f"}
Sep 17 23:07:40.600102 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.600070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njwnq" event={"ID":"f6a51291-55ed-41ea-8029-bf5b677956a8","Type":"ContainerStarted","Data":"6e0b142dfee7123c30087ba6c23d3ea4170370dd6734e3af6a4f41d6509d9f62"}
Sep 17 23:07:40.600219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.600127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njwnq" event={"ID":"f6a51291-55ed-41ea-8029-bf5b677956a8","Type":"ContainerStarted","Data":"534f53f0c65f2a634b13084ef7dcd8154d9cf536ea843795f08c26877f494ecd"}
Sep 17 23:07:40.617418 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.617359    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-6f4ms" podStartSLOduration=2.978391969 podStartE2EDuration="4.617342474s" totalImagesPullingTime="1.638950505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:40.616837935 +0000 UTC m=+71.046697753" watchObservedRunningTime="2026-09-17 23:07:40.617342474 +0000 UTC m=+71.047202291"
Sep 17 23:07:40.636198 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.636132    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2bqjt" podStartSLOduration=2.6918785229999997 podStartE2EDuration="4.636114312s" totalImagesPullingTime="1.944235789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:40.635343173 +0000 UTC m=+71.065202993" watchObservedRunningTime="2026-09-17 23:07:40.636114312 +0000 UTC m=+71.065974130"
Sep 17 23:07:40.652530 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.652476    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-njwnq" podStartSLOduration=68.471824895 podStartE2EDuration="1m10.652460074s" totalImagesPullingTime="2.180635179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:40.652175437 +0000 UTC m=+71.082035254" watchObservedRunningTime="2026-09-17 23:07:40.652460074 +0000 UTC m=+71.082319892"
Sep 17 23:07:40.757345 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.757309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:40.853258 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:40.853143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.027002 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.026963    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-678485c766-5b9fk"]
Sep 17 23:07:41.030302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.030278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.032678 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.032641    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 23:07:41.033467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.033206    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:07:41.033467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.033242    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 23:07:41.033467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.033306    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 23:07:41.033664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.033400    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9crv2\""
Sep 17 23:07:41.033664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.033552    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-27j5v7meajcb8\""
Sep 17 23:07:41.044728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.042771    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-678485c766-5b9fk"]
Sep 17 23:07:41.141239 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.141200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-secret-metrics-server-client-certs\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.141485 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.141246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9b7543c-2bc8-46b8-af17-56d1e6be1294-metrics-server-audit-profiles\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.141485 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.141290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9b7543c-2bc8-46b8-af17-56d1e6be1294-audit-log\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.141485 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.141320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b7543c-2bc8-46b8-af17-56d1e6be1294-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.141485 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.141346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctx2\" (UniqueName: \"kubernetes.io/projected/a9b7543c-2bc8-46b8-af17-56d1e6be1294-kube-api-access-cctx2\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.141485 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.141368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-secret-metrics-server-tls\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.141485 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.141389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-client-ca-bundle\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.242645 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.242604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-secret-metrics-server-tls\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.242858 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.242670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-client-ca-bundle\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.242858 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.242749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-secret-metrics-server-client-certs\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.242858 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.242775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9b7543c-2bc8-46b8-af17-56d1e6be1294-metrics-server-audit-profiles\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.242858 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.242847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9b7543c-2bc8-46b8-af17-56d1e6be1294-audit-log\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.243027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.242876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b7543c-2bc8-46b8-af17-56d1e6be1294-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.243027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.242910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctx2\" (UniqueName: \"kubernetes.io/projected/a9b7543c-2bc8-46b8-af17-56d1e6be1294-kube-api-access-cctx2\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.243590 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.243561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9b7543c-2bc8-46b8-af17-56d1e6be1294-audit-log\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.243875 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.243849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b7543c-2bc8-46b8-af17-56d1e6be1294-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.244227 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.244161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9b7543c-2bc8-46b8-af17-56d1e6be1294-metrics-server-audit-profiles\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.246587 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.246557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-client-ca-bundle\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.246676 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.246557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-secret-metrics-server-tls\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.248084 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.248062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9b7543c-2bc8-46b8-af17-56d1e6be1294-secret-metrics-server-client-certs\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.251483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.251460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctx2\" (UniqueName: \"kubernetes.io/projected/a9b7543c-2bc8-46b8-af17-56d1e6be1294-kube-api-access-cctx2\") pod \"metrics-server-678485c766-5b9fk\" (UID: \"a9b7543c-2bc8-46b8-af17-56d1e6be1294\") " pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.343105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.343065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:07:41.475286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.475215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.481991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.481961    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"]
Sep 17 23:07:41.485350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.485331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.488561 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.488517    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 23:07:41.488719 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.488523    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9ww5f\""
Sep 17 23:07:41.496000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.495975    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"]
Sep 17 23:07:41.545183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.545140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vkbhw\" (UID: \"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.545362 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.545207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47sfb\" (UniqueName: \"kubernetes.io/projected/a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef-kube-api-access-47sfb\") pod \"monitoring-plugin-5984459ff8-vkbhw\" (UID: \"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.573575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.573531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.632437 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.632401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.646557 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.646525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vkbhw\" (UID: \"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.646721 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.646634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47sfb\" (UniqueName: \"kubernetes.io/projected/a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef-kube-api-access-47sfb\") pod \"monitoring-plugin-5984459ff8-vkbhw\" (UID: \"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.649436 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.649414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vkbhw\" (UID: \"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.655878 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.655851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47sfb\" (UniqueName: \"kubernetes.io/projected/a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef-kube-api-access-47sfb\") pod \"monitoring-plugin-5984459ff8-vkbhw\" (UID: \"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.791257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.791168    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f5c889b8-rx749"]
Sep 17 23:07:41.796953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.796923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:41.824949 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.824911    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9866ccdd-gmkkt"]
Sep 17 23:07:41.828409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.828079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:41.840440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.840416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9866ccdd-gmkkt"]
Sep 17 23:07:41.949723 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.949686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-trusted-ca-bundle\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:41.949917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.949726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtln\" (UniqueName: \"kubernetes.io/projected/125a30ca-997a-4893-b703-a1aeef9ea328-kube-api-access-tjtln\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:41.949917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.949836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-serving-cert\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:41.949917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.949864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-oauth-serving-cert\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:41.949917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.949886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-oauth-config\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:41.949917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.949901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-service-ca\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:41.950129 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:41.949961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-console-config\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.050718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.050634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-trusted-ca-bundle\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.050718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.050675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtln\" (UniqueName: \"kubernetes.io/projected/125a30ca-997a-4893-b703-a1aeef9ea328-kube-api-access-tjtln\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.050718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.050707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-serving-cert\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.051009 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.050734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-oauth-serving-cert\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.051083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.051065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-oauth-config\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.051137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.051103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-service-ca\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.051137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.051128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-console-config\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.051587 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.051560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-oauth-serving-cert\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.051710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.051599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-trusted-ca-bundle\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.051770 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.051747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-console-config\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.052137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.052119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-service-ca\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.053744 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.053723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-serving-cert\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.053859 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.053743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-oauth-config\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.060160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.060119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtln\" (UniqueName: \"kubernetes.io/projected/125a30ca-997a-4893-b703-a1aeef9ea328-kube-api-access-tjtln\") pod \"console-9866ccdd-gmkkt\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") " pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.099783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.099735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:42.140117 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.140081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:42.199368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.199323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:42.272585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.272540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:07:42.974710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.974671    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:07:42.975231 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.974860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:42.979568 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.979543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:42.986381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.985420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:07:42.986381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.985575    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:07:42.986381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.985830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:07:42.986381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.986029    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:07:42.986381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.986046    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:07:42.986381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.986053    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:07:42.986381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.986179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:07:42.986853 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.986493    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:07:42.986853 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.986715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rr5cr29js559\""
Sep 17 23:07:42.987302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.986973    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:07:42.987302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.987121    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5w8gr\""
Sep 17 23:07:42.987302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.987201    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:07:42.987767 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.987531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:07:42.991320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:42.987535    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:07:43.005231 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.005149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:07:43.006079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.006054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:07:43.009961 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.009938    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:07:43.030716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.030643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9866ccdd-gmkkt"]
Sep 17 23:07:43.036455 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:43.036407    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125a30ca_997a_4893_b703_a1aeef9ea328.slice/crio-5bfad8fc2ee7a02e08374b12285d3b5264b954ca909ad7f4392c628ab1b3b064 WatchSource:0}: Error finding container 5bfad8fc2ee7a02e08374b12285d3b5264b954ca909ad7f4392c628ab1b3b064: Status 404 returned error can't find the container with id 5bfad8fc2ee7a02e08374b12285d3b5264b954ca909ad7f4392c628ab1b3b064
Sep 17 23:07:43.047968 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.045549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-678485c766-5b9fk"]
Sep 17 23:07:43.048337 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:43.048308    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b7543c_2bc8_46b8_af17_56d1e6be1294.slice/crio-ba5ed048330b1d4f01b913668c0d01d1b1dc49c54274aa741b67febafb5d4177 WatchSource:0}: Error finding container ba5ed048330b1d4f01b913668c0d01d1b1dc49c54274aa741b67febafb5d4177: Status 404 returned error can't find the container with id ba5ed048330b1d4f01b913668c0d01d1b1dc49c54274aa741b67febafb5d4177
Sep 17 23:07:43.048686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.048662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:43.058989 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.057671    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"]
Sep 17 23:07:43.060034 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060207 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060259 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060338 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-config\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9xc\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-kube-api-access-5s9xc\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.060830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.061016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.061016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.061016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.061016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.061016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.060950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.062126 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:43.062083    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b66e0b_4b3a_4c05_8536_37bb98f0c9ef.slice/crio-e198c8f4ea03cd26f70d0d9048212ba78821fbe337db144c6c601f12f7f24560 WatchSource:0}: Error finding container e198c8f4ea03cd26f70d0d9048212ba78821fbe337db144c6c601f12f7f24560: Status 404 returned error can't find the container with id e198c8f4ea03cd26f70d0d9048212ba78821fbe337db144c6c601f12f7f24560
Sep 17 23:07:43.161672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-config\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9xc\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-kube-api-access-5s9xc\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.161944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.161987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.162005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.162046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.162075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.162101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.162125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.162423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.162160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.163411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.163725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.164237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.164934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.165176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.165730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.165771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-config\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.165990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.166156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.166263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.166474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.166687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.166644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.167351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.166883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.169854 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.169786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.170618 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.170593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.170913 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.170888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.171341 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.171313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.171587 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.171570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.173367 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.173337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.173829 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.173778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9xc\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-kube-api-access-5s9xc\") pod \"prometheus-k8s-0\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.309654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.309565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:43.599624 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.599544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:07:43.605088 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:07:43.605054    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1846824_0aca_42ae_b176_1da0d260c040.slice/crio-9b08a9180030f6217008f7d34d294ce9f46464b5efcd1cdc5f45d451ff8e084a WatchSource:0}: Error finding container 9b08a9180030f6217008f7d34d294ce9f46464b5efcd1cdc5f45d451ff8e084a: Status 404 returned error can't find the container with id 9b08a9180030f6217008f7d34d294ce9f46464b5efcd1cdc5f45d451ff8e084a
Sep 17 23:07:43.615570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.615536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zlv7c" event={"ID":"26053f12-63eb-48b5-a610-95ce5cd2c941","Type":"ContainerStarted","Data":"c49f50788e626246396a25d8412a5898795c2c233bedf71c45da12716b9112c8"}
Sep 17 23:07:43.617323 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.617287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9866ccdd-gmkkt" event={"ID":"125a30ca-997a-4893-b703-a1aeef9ea328","Type":"ContainerStarted","Data":"b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae"}
Sep 17 23:07:43.617440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.617342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9866ccdd-gmkkt" event={"ID":"125a30ca-997a-4893-b703-a1aeef9ea328","Type":"ContainerStarted","Data":"5bfad8fc2ee7a02e08374b12285d3b5264b954ca909ad7f4392c628ab1b3b064"}
Sep 17 23:07:43.618622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.618598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw" event={"ID":"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef","Type":"ContainerStarted","Data":"e198c8f4ea03cd26f70d0d9048212ba78821fbe337db144c6c601f12f7f24560"}
Sep 17 23:07:43.620436 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.620403    2580 generic.go:356] "Generic (PLEG): container finished" podID="8fc37975-cb27-4d09-9c60-916d3a88a503" containerID="843d760d71d3af244539b16280c798257baec23abed85ca389b137eada8419ab" exitCode=0
Sep 17 23:07:43.620537 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.620498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbtkm" event={"ID":"8fc37975-cb27-4d09-9c60-916d3a88a503","Type":"ContainerDied","Data":"843d760d71d3af244539b16280c798257baec23abed85ca389b137eada8419ab"}
Sep 17 23:07:43.622071 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.622047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-678485c766-5b9fk" event={"ID":"a9b7543c-2bc8-46b8-af17-56d1e6be1294","Type":"ContainerStarted","Data":"ba5ed048330b1d4f01b913668c0d01d1b1dc49c54274aa741b67febafb5d4177"}
Sep 17 23:07:43.624213 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.623841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xrmrc" event={"ID":"8fc02dbe-b98b-4ab3-9fe5-fcbe41986d8e","Type":"ContainerStarted","Data":"a597bd77bcb3182c69d9eed8ec2e43fa30fa7c1a269d80a78baef8e45df5f01f"}
Sep 17 23:07:43.624213 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.623909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:07:43.625445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.625426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerStarted","Data":"9b08a9180030f6217008f7d34d294ce9f46464b5efcd1cdc5f45d451ff8e084a"}
Sep 17 23:07:43.631383 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.631346    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zlv7c" podStartSLOduration=68.31635526 podStartE2EDuration="1m13.631332862s" totalImagesPullingTime="5.314977602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:43.62944927 +0000 UTC m=+74.059309089" watchObservedRunningTime="2026-09-17 23:07:43.631332862 +0000 UTC m=+74.061192726"
Sep 17 23:07:43.685391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.685328    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9866ccdd-gmkkt" podStartSLOduration=2.685305836 podStartE2EDuration="2.685305836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:43.667081943 +0000 UTC m=+74.096941763" watchObservedRunningTime="2026-09-17 23:07:43.685305836 +0000 UTC m=+74.115165656"
Sep 17 23:07:43.686074 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:43.686034    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xrmrc" podStartSLOduration=68.391936602 podStartE2EDuration="1m13.686022007s" totalImagesPullingTime="5.294085405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:43.684787541 +0000 UTC m=+74.114647360" watchObservedRunningTime="2026-09-17 23:07:43.686022007 +0000 UTC m=+74.115881830"
Sep 17 23:07:44.632998 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:44.632511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbtkm" event={"ID":"8fc37975-cb27-4d09-9c60-916d3a88a503","Type":"ContainerStarted","Data":"ee4118ad8f6a56ca111df34b2b04561cb2d2bf0c02113abefd34b623f10c71ed"}
Sep 17 23:07:44.632998 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:44.632588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rbtkm" event={"ID":"8fc37975-cb27-4d09-9c60-916d3a88a503","Type":"ContainerStarted","Data":"a651f9040e063511f85d6c279982b5df070b3111281a8c3e42001a64733c20dc"}
Sep 17 23:07:44.653908 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:44.653857    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rbtkm" podStartSLOduration=5.343269545 podStartE2EDuration="8.653841711s" totalImagesPullingTime="3.1965809s" totalInitContainerRuntime="113.991266ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:44.652444083 +0000 UTC m=+75.082303901" watchObservedRunningTime="2026-09-17 23:07:44.653841711 +0000 UTC m=+75.083701528"
Sep 17 23:07:45.583358 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.583325    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9866ccdd-gmkkt"]
Sep 17 23:07:45.638536 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.638499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-678485c766-5b9fk" event={"ID":"a9b7543c-2bc8-46b8-af17-56d1e6be1294","Type":"ContainerStarted","Data":"ca7a4e0c4663b5d0f0ef3e26d1d1a9bd4b637953eb3e4a08f195cfd4f0940fcf"}
Sep 17 23:07:45.640196 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.640162    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1846824-0aca-42ae-b176-1da0d260c040" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74" exitCode=0
Sep 17 23:07:45.640327 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.640241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"}
Sep 17 23:07:45.642088 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.642063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw" event={"ID":"a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef","Type":"ContainerStarted","Data":"2e5b675de211e46958509306f8bfd5e346bdef1e8923f6b72a6b9b7d237383ca"}
Sep 17 23:07:45.642398 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.642374    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:45.648530 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.648502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw"
Sep 17 23:07:45.656764 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.656719    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-678485c766-5b9fk" podStartSLOduration=2.8358156169999997 podStartE2EDuration="4.656702377s" totalImagesPullingTime="1.82088676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:45.655668649 +0000 UTC m=+76.085528468" watchObservedRunningTime="2026-09-17 23:07:45.656702377 +0000 UTC m=+76.086562196"
Sep 17 23:07:45.676514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:45.676461    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vkbhw" podStartSLOduration=2.868748518 podStartE2EDuration="4.676447441s" totalImagesPullingTime="1.807698923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:45.674590657 +0000 UTC m=+76.104450475" watchObservedRunningTime="2026-09-17 23:07:45.676447441 +0000 UTC m=+76.106307263"
Sep 17 23:07:46.458098 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:46.458056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:46.557952 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:46.557918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:46.655076 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:46.655049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:46.718133 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:46.718044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:49.662535 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:49.662505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerStarted","Data":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"}
Sep 17 23:07:49.662942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:49.662568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerStarted","Data":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"}
Sep 17 23:07:51.157915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:51.157881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:51.255086 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:51.255046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:51.676775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:51.676739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerStarted","Data":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"}
Sep 17 23:07:51.676964 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:51.676842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerStarted","Data":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"}
Sep 17 23:07:51.676964 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:51.676882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerStarted","Data":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"}
Sep 17 23:07:51.676964 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:51.676919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerStarted","Data":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"}
Sep 17 23:07:51.705697 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:51.705631    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6234423590000002 podStartE2EDuration="9.705609441s" totalImagesPullingTime="7.060801296s" totalInitContainerRuntime="21.365786ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:51.703059124 +0000 UTC m=+82.132918942" watchObservedRunningTime="2026-09-17 23:07:51.705609441 +0000 UTC m=+82.135469264"
Sep 17 23:07:52.140269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:52.140240    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:07:53.309918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:53.309869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:55.656727 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:55.656687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:55.715195 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:55.715158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:56.653059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:56.653024    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:56.753595 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:56.753554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:56.853654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:07:56.853619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:01.344121 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:01.344079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:08:01.344569 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:01.344187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:08:06.818347 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:06.818286    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58f5c889b8-rx749" podUID="32c5b3f7-46a2-49a4-9464-84168ea9c74d" containerName="console" containerID="cri-o://6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850" gracePeriod=15
Sep 17 23:08:07.584149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.584108    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:08:07.586689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.586664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-serving-cert\") pod \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") "
Sep 17 23:08:07.586866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.586702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-service-ca\") pod \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") "
Sep 17 23:08:07.586866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.586737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl74z\" (UniqueName: \"kubernetes.io/projected/32c5b3f7-46a2-49a4-9464-84168ea9c74d-kube-api-access-hl74z\") pod \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") "
Sep 17 23:08:07.586866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.586755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-oauth-serving-cert\") pod \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") "
Sep 17 23:08:07.586866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.586784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-oauth-config\") pod \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") "
Sep 17 23:08:07.586866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.586817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-trusted-ca-bundle\") pod \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") "
Sep 17 23:08:07.586866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.586843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-config\") pod \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\" (UID: \"32c5b3f7-46a2-49a4-9464-84168ea9c74d\") "
Sep 17 23:08:07.587285 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.587257    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "32c5b3f7-46a2-49a4-9464-84168ea9c74d" (UID: "32c5b3f7-46a2-49a4-9464-84168ea9c74d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:07.587359 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.587319    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-service-ca" (OuterVolumeSpecName: "service-ca") pod "32c5b3f7-46a2-49a4-9464-84168ea9c74d" (UID: "32c5b3f7-46a2-49a4-9464-84168ea9c74d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:07.587951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.587918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-config" (OuterVolumeSpecName: "console-config") pod "32c5b3f7-46a2-49a4-9464-84168ea9c74d" (UID: "32c5b3f7-46a2-49a4-9464-84168ea9c74d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:07.587951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.587938    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "32c5b3f7-46a2-49a4-9464-84168ea9c74d" (UID: "32c5b3f7-46a2-49a4-9464-84168ea9c74d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:07.589381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.589353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "32c5b3f7-46a2-49a4-9464-84168ea9c74d" (UID: "32c5b3f7-46a2-49a4-9464-84168ea9c74d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:08:07.589487 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.589430    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "32c5b3f7-46a2-49a4-9464-84168ea9c74d" (UID: "32c5b3f7-46a2-49a4-9464-84168ea9c74d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:08:07.589811 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.589775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c5b3f7-46a2-49a4-9464-84168ea9c74d-kube-api-access-hl74z" (OuterVolumeSpecName: "kube-api-access-hl74z") pod "32c5b3f7-46a2-49a4-9464-84168ea9c74d" (UID: "32c5b3f7-46a2-49a4-9464-84168ea9c74d"). InnerVolumeSpecName "kube-api-access-hl74z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:08:07.688313 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.688273    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-oauth-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:07.688313 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.688306    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-trusted-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:07.688313 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.688316    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:07.688313 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.688325    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c5b3f7-46a2-49a4-9464-84168ea9c74d-console-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:07.688583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.688335    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:07.688583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.688344    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl74z\" (UniqueName: \"kubernetes.io/projected/32c5b3f7-46a2-49a4-9464-84168ea9c74d-kube-api-access-hl74z\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:07.688583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.688353    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c5b3f7-46a2-49a4-9464-84168ea9c74d-oauth-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:07.744302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.744200    2580 generic.go:356] "Generic (PLEG): container finished" podID="32c5b3f7-46a2-49a4-9464-84168ea9c74d" containerID="6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850" exitCode=0
Sep 17 23:08:07.744302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.744266    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f5c889b8-rx749"
Sep 17 23:08:07.744515 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.744295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f5c889b8-rx749" event={"ID":"32c5b3f7-46a2-49a4-9464-84168ea9c74d","Type":"ContainerDied","Data":"6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850"}
Sep 17 23:08:07.744515 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.744387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f5c889b8-rx749" event={"ID":"32c5b3f7-46a2-49a4-9464-84168ea9c74d","Type":"ContainerDied","Data":"d2136faa6a54e476fc9f90eca080e0cd80f0c21c65c6ab0da278250dfa686fc9"}
Sep 17 23:08:07.744515 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.744416    2580 scope.go:117] "RemoveContainer" containerID="6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850"
Sep 17 23:08:07.753589 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.753566    2580 scope.go:117] "RemoveContainer" containerID="6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850"
Sep 17 23:08:07.753890 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:08:07.753869    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850\": container with ID starting with 6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850 not found: ID does not exist" containerID="6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850"
Sep 17 23:08:07.753950 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.753901    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850"} err="failed to get container status \"6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850\": rpc error: code = NotFound desc = could not find container \"6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850\": container with ID starting with 6c12ca8c2a3a7ec43e9514741cc1e88d2e7019b4939e6cc810bbd704fb242850 not found: ID does not exist"
Sep 17 23:08:07.766656 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.766628    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f5c889b8-rx749"]
Sep 17 23:08:07.770616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:07.770587    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58f5c889b8-rx749"]
Sep 17 23:08:08.106919 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:08.106830    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c5b3f7-46a2-49a4-9464-84168ea9c74d" path="/var/lib/kubelet/pods/32c5b3f7-46a2-49a4-9464-84168ea9c74d/volumes"
Sep 17 23:08:10.662647 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:10.662590    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9866ccdd-gmkkt" podUID="125a30ca-997a-4893-b703-a1aeef9ea328" containerName="console" containerID="cri-o://b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae" gracePeriod=15
Sep 17 23:08:10.922903 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:10.922818    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:08:11.018875 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.018837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-oauth-serving-cert\") pod \"125a30ca-997a-4893-b703-a1aeef9ea328\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") "
Sep 17 23:08:11.019033 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.018890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjtln\" (UniqueName: \"kubernetes.io/projected/125a30ca-997a-4893-b703-a1aeef9ea328-kube-api-access-tjtln\") pod \"125a30ca-997a-4893-b703-a1aeef9ea328\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") "
Sep 17 23:08:11.019033 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.018926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-serving-cert\") pod \"125a30ca-997a-4893-b703-a1aeef9ea328\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") "
Sep 17 23:08:11.019157 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-console-config\") pod \"125a30ca-997a-4893-b703-a1aeef9ea328\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") "
Sep 17 23:08:11.019211 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019165    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-trusted-ca-bundle\") pod \"125a30ca-997a-4893-b703-a1aeef9ea328\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") "
Sep 17 23:08:11.019211 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-oauth-config\") pod \"125a30ca-997a-4893-b703-a1aeef9ea328\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") "
Sep 17 23:08:11.019312 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-service-ca\") pod \"125a30ca-997a-4893-b703-a1aeef9ea328\" (UID: \"125a30ca-997a-4893-b703-a1aeef9ea328\") "
Sep 17 23:08:11.019440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "125a30ca-997a-4893-b703-a1aeef9ea328" (UID: "125a30ca-997a-4893-b703-a1aeef9ea328"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:11.019621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-console-config" (OuterVolumeSpecName: "console-config") pod "125a30ca-997a-4893-b703-a1aeef9ea328" (UID: "125a30ca-997a-4893-b703-a1aeef9ea328"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:11.019906 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-service-ca" (OuterVolumeSpecName: "service-ca") pod "125a30ca-997a-4893-b703-a1aeef9ea328" (UID: "125a30ca-997a-4893-b703-a1aeef9ea328"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:11.020018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.019867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "125a30ca-997a-4893-b703-a1aeef9ea328" (UID: "125a30ca-997a-4893-b703-a1aeef9ea328"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:11.021333 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.021304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "125a30ca-997a-4893-b703-a1aeef9ea328" (UID: "125a30ca-997a-4893-b703-a1aeef9ea328"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:08:11.021429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.021339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125a30ca-997a-4893-b703-a1aeef9ea328-kube-api-access-tjtln" (OuterVolumeSpecName: "kube-api-access-tjtln") pod "125a30ca-997a-4893-b703-a1aeef9ea328" (UID: "125a30ca-997a-4893-b703-a1aeef9ea328"). InnerVolumeSpecName "kube-api-access-tjtln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:08:11.021429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.021413    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "125a30ca-997a-4893-b703-a1aeef9ea328" (UID: "125a30ca-997a-4893-b703-a1aeef9ea328"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:08:11.120164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.120129    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:11.120164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.120158    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-console-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:11.120164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.120168    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-trusted-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:11.120164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.120176    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/125a30ca-997a-4893-b703-a1aeef9ea328-console-oauth-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:11.120415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.120186    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:11.120415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.120195    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/125a30ca-997a-4893-b703-a1aeef9ea328-oauth-serving-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:11.120415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.120203    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjtln\" (UniqueName: \"kubernetes.io/projected/125a30ca-997a-4893-b703-a1aeef9ea328-kube-api-access-tjtln\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:08:11.189167 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.189108    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bt4qj_c45bc469-07af-45ce-bfb0-aa730901da10/serve-healthcheck-canary/0.log"
Sep 17 23:08:11.761985 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.761944    2580 generic.go:356] "Generic (PLEG): container finished" podID="125a30ca-997a-4893-b703-a1aeef9ea328" containerID="b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae" exitCode=0
Sep 17 23:08:11.762417 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.761995    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9866ccdd-gmkkt"
Sep 17 23:08:11.762417 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.762007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9866ccdd-gmkkt" event={"ID":"125a30ca-997a-4893-b703-a1aeef9ea328","Type":"ContainerDied","Data":"b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae"}
Sep 17 23:08:11.762417 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.762057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9866ccdd-gmkkt" event={"ID":"125a30ca-997a-4893-b703-a1aeef9ea328","Type":"ContainerDied","Data":"5bfad8fc2ee7a02e08374b12285d3b5264b954ca909ad7f4392c628ab1b3b064"}
Sep 17 23:08:11.762417 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.762083    2580 scope.go:117] "RemoveContainer" containerID="b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae"
Sep 17 23:08:11.772788 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.772758    2580 scope.go:117] "RemoveContainer" containerID="b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae"
Sep 17 23:08:11.773645 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:08:11.773418    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae\": container with ID starting with b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae not found: ID does not exist" containerID="b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae"
Sep 17 23:08:11.773645 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.773490    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae"} err="failed to get container status \"b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae\": rpc error: code = NotFound desc = could not find container \"b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae\": container with ID starting with b0f20247f081e84b05c080fb699722630c8ba937cf29f6d99e43b266cc3fe1ae not found: ID does not exist"
Sep 17 23:08:11.786702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.786677    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9866ccdd-gmkkt"]
Sep 17 23:08:11.790087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:11.790065    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9866ccdd-gmkkt"]
Sep 17 23:08:12.097448 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:12.097364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:12.107045 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:12.107009    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125a30ca-997a-4893-b703-a1aeef9ea328" path="/var/lib/kubelet/pods/125a30ca-997a-4893-b703-a1aeef9ea328/volumes"
Sep 17 23:08:12.199056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:12.199006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:14.635402 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:14.635372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xrmrc"
Sep 17 23:08:20.099500 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:20.099467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:20.206164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:20.206125    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:20.269813 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:20.269768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:21.348888 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:21.348856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:08:21.353573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:21.353545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-678485c766-5b9fk"
Sep 17 23:08:43.310147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:43.310095    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:08:43.329480 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:43.329456    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:08:43.904513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:43.904474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:08:52.777614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:52.777583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:52.846495 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:08:52.846453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:01.322629 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.322590    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:09:01.323257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.322910    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="prometheus" containerID="cri-o://913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2" gracePeriod=600
Sep 17 23:09:01.323257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.322972    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy" containerID="cri-o://c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26" gracePeriod=600
Sep 17 23:09:01.323257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.322974    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-web" containerID="cri-o://f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761" gracePeriod=600
Sep 17 23:09:01.323257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.322970    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="thanos-sidecar" containerID="cri-o://821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944" gracePeriod=600
Sep 17 23:09:01.323257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.323066    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="config-reloader" containerID="cri-o://5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1" gracePeriod=600
Sep 17 23:09:01.323257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.323138    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460" gracePeriod=600
Sep 17 23:09:01.566445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.566421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:01.740910 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.740872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-0\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.740910 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.740915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-kube-rbac-proxy\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.740940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-config-out\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.740961    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-1\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.740986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s9xc\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-kube-api-access-5s9xc\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-web-config\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-metrics-client-certs\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-grpc-tls\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741232    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-metrics-client-ca\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-tls\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741309    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:01.741350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-trusted-ca-bundle\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-2\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-serving-certs-ca-bundle\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-kubelet-serving-ca-bundle\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-thanos-prometheus-http-client-file\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-config\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-tls-assets\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-db\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.741684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b1846824-0aca-42ae-b176-1da0d260c040\" (UID: \"b1846824-0aca-42ae-b176-1da0d260c040\") "
Sep 17 23:09:01.742093 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.741872    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.742333 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.742304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:01.742538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.742351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:01.742737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.742712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:01.744030 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.743997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-config-out" (OuterVolumeSpecName: "config-out") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:09:01.744030 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.744019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-kube-api-access-5s9xc" (OuterVolumeSpecName: "kube-api-access-5s9xc") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "kube-api-access-5s9xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:09:01.744405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.744378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:01.744702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.744673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.744784 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.744746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.744886 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.744789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:01.745265 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.745059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.745265 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.745092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.745423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.745316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-config" (OuterVolumeSpecName: "config") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.745617 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.745585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:01.745919 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.745900    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:09:01.746790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.746771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:09:01.747314 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.747292    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.747395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.747358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.747462 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.747424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.754598 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.754572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-web-config" (OuterVolumeSpecName: "web-config") pod "b1846824-0aca-42ae-b176-1da0d260c040" (UID: "b1846824-0aca-42ae-b176-1da0d260c040"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:01.842478 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842443    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-metrics-client-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842478 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842473    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-grpc-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842478 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842484    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842493    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-metrics-client-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842502    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842510    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842518    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842527    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842535    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842544    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842552    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842560    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-tls-assets\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842568    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-db\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842576    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842586    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1846824-0aca-42ae-b176-1da0d260c040-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842594    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-secret-kube-rbac-proxy\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842604    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1846824-0aca-42ae-b176-1da0d260c040-config-out\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842614    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s9xc\" (UniqueName: \"kubernetes.io/projected/b1846824-0aca-42ae-b176-1da0d260c040-kube-api-access-5s9xc\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.842684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.842641    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1846824-0aca-42ae-b176-1da0d260c040-web-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:09:01.955675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955642    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1846824-0aca-42ae-b176-1da0d260c040" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460" exitCode=0
Sep 17 23:09:01.955675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955665    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1846824-0aca-42ae-b176-1da0d260c040" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26" exitCode=0
Sep 17 23:09:01.955675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955671    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1846824-0aca-42ae-b176-1da0d260c040" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761" exitCode=0
Sep 17 23:09:01.955675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955676    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1846824-0aca-42ae-b176-1da0d260c040" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944" exitCode=0
Sep 17 23:09:01.955675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955681    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1846824-0aca-42ae-b176-1da0d260c040" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1" exitCode=0
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955687    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1846824-0aca-42ae-b176-1da0d260c040" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2" exitCode=0
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955712    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"}
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"}
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955824    2580 scope.go:117] "RemoveContainer" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"}
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"}
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"}
Sep 17 23:09:01.955953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"}
Sep 17 23:09:01.956325 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.955985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1846824-0aca-42ae-b176-1da0d260c040","Type":"ContainerDied","Data":"9b08a9180030f6217008f7d34d294ce9f46464b5efcd1cdc5f45d451ff8e084a"}
Sep 17 23:09:01.966068 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.966043    2580 scope.go:117] "RemoveContainer" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:01.973727 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.973709    2580 scope.go:117] "RemoveContainer" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:01.980988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.980663    2580 scope.go:117] "RemoveContainer" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:01.982347 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.982328    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:09:01.985952 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.985931    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:09:01.988499 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.988482    2580 scope.go:117] "RemoveContainer" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:01.995758 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:01.995740    2580 scope.go:117] "RemoveContainer" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.003779 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.003760    2580 scope.go:117] "RemoveContainer" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.011699 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.011684    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:09:02.011755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.011729    2580 scope.go:117] "RemoveContainer" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:02.012056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012041    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="init-config-reloader"
Sep 17 23:09:02.012100 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012060    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="init-config-reloader"
Sep 17 23:09:02.012100 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:09:02.012062    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": container with ID starting with 5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460 not found: ID does not exist" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:02.012160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012092    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"} err="failed to get container status \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": rpc error: code = NotFound desc = could not find container \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": container with ID starting with 5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460 not found: ID does not exist"
Sep 17 23:09:02.012160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012110    2580 scope.go:117] "RemoveContainer" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:02.012160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012069    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="prometheus"
Sep 17 23:09:02.012160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012151    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="prometheus"
Sep 17 23:09:02.012160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012159    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="config-reloader"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012165    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="config-reloader"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012175    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012180    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012191    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125a30ca-997a-4893-b703-a1aeef9ea328" containerName="console"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012196    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="125a30ca-997a-4893-b703-a1aeef9ea328" containerName="console"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012204    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-web"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012210    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-web"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012219    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c5b3f7-46a2-49a4-9464-84168ea9c74d" containerName="console"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012228    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c5b3f7-46a2-49a4-9464-84168ea9c74d" containerName="console"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012241    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="thanos-sidecar"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012246    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="thanos-sidecar"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012253    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-thanos"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012257    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-thanos"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012302    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c5b3f7-46a2-49a4-9464-84168ea9c74d" containerName="console"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012313    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="125a30ca-997a-4893-b703-a1aeef9ea328" containerName="console"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012320    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-web"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012328    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="thanos-sidecar"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012335    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy"
Sep 17 23:09:02.012342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012342    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="prometheus"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012352    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="config-reloader"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012359    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1846824-0aca-42ae-b176-1da0d260c040" containerName="kube-rbac-proxy-thanos"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:09:02.012355    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": container with ID starting with c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26 not found: ID does not exist" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012377    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"} err="failed to get container status \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": rpc error: code = NotFound desc = could not find container \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": container with ID starting with c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26 not found: ID does not exist"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012389    2580 scope.go:117] "RemoveContainer" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:09:02.012839    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": container with ID starting with f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761 not found: ID does not exist" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012870    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"} err="failed to get container status \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": rpc error: code = NotFound desc = could not find container \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": container with ID starting with f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761 not found: ID does not exist"
Sep 17 23:09:02.013247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.012896    2580 scope.go:117] "RemoveContainer" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:02.013545 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:09:02.013426    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": container with ID starting with 821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944 not found: ID does not exist" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:02.013545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.013452    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"} err="failed to get container status \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": rpc error: code = NotFound desc = could not find container \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": container with ID starting with 821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944 not found: ID does not exist"
Sep 17 23:09:02.013545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.013472    2580 scope.go:117] "RemoveContainer" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:02.013752 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:09:02.013731    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": container with ID starting with 5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1 not found: ID does not exist" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:02.013862 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.013758    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"} err="failed to get container status \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": rpc error: code = NotFound desc = could not find container \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": container with ID starting with 5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1 not found: ID does not exist"
Sep 17 23:09:02.013862 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.013772    2580 scope.go:117] "RemoveContainer" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.014072 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:09:02.014050    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": container with ID starting with 913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2 not found: ID does not exist" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.014153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014078    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"} err="failed to get container status \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": rpc error: code = NotFound desc = could not find container \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": container with ID starting with 913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2 not found: ID does not exist"
Sep 17 23:09:02.014153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014093    2580 scope.go:117] "RemoveContainer" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.014358 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:09:02.014337    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": container with ID starting with 74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74 not found: ID does not exist" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.014425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014362    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"} err="failed to get container status \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": rpc error: code = NotFound desc = could not find container \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": container with ID starting with 74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74 not found: ID does not exist"
Sep 17 23:09:02.014425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014378    2580 scope.go:117] "RemoveContainer" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:02.014663 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014644    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"} err="failed to get container status \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": rpc error: code = NotFound desc = could not find container \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": container with ID starting with 5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460 not found: ID does not exist"
Sep 17 23:09:02.014734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014663    2580 scope.go:117] "RemoveContainer" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:02.014951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014932    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"} err="failed to get container status \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": rpc error: code = NotFound desc = could not find container \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": container with ID starting with c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26 not found: ID does not exist"
Sep 17 23:09:02.015013 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.014951    2580 scope.go:117] "RemoveContainer" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:02.015210 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015191    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"} err="failed to get container status \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": rpc error: code = NotFound desc = could not find container \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": container with ID starting with f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761 not found: ID does not exist"
Sep 17 23:09:02.015257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015212    2580 scope.go:117] "RemoveContainer" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:02.015450 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015431    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"} err="failed to get container status \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": rpc error: code = NotFound desc = could not find container \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": container with ID starting with 821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944 not found: ID does not exist"
Sep 17 23:09:02.015492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015452    2580 scope.go:117] "RemoveContainer" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:02.015689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015673    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"} err="failed to get container status \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": rpc error: code = NotFound desc = could not find container \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": container with ID starting with 5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1 not found: ID does not exist"
Sep 17 23:09:02.015689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015688    2580 scope.go:117] "RemoveContainer" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.015944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015897    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"} err="failed to get container status \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": rpc error: code = NotFound desc = could not find container \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": container with ID starting with 913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2 not found: ID does not exist"
Sep 17 23:09:02.015944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.015921    2580 scope.go:117] "RemoveContainer" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.016146 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.016129    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"} err="failed to get container status \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": rpc error: code = NotFound desc = could not find container \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": container with ID starting with 74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74 not found: ID does not exist"
Sep 17 23:09:02.016146 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.016145    2580 scope.go:117] "RemoveContainer" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:02.016362 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.016341    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"} err="failed to get container status \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": rpc error: code = NotFound desc = could not find container \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": container with ID starting with 5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460 not found: ID does not exist"
Sep 17 23:09:02.016362 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.016358    2580 scope.go:117] "RemoveContainer" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:02.016518 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.016504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.016602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.016580    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"} err="failed to get container status \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": rpc error: code = NotFound desc = could not find container \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": container with ID starting with c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26 not found: ID does not exist"
Sep 17 23:09:02.016642 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.016604    2580 scope.go:117] "RemoveContainer" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:02.017149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017120    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"} err="failed to get container status \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": rpc error: code = NotFound desc = could not find container \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": container with ID starting with f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761 not found: ID does not exist"
Sep 17 23:09:02.017237 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017150    2580 scope.go:117] "RemoveContainer" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:02.017388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017363    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"} err="failed to get container status \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": rpc error: code = NotFound desc = could not find container \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": container with ID starting with 821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944 not found: ID does not exist"
Sep 17 23:09:02.017388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017389    2580 scope.go:117] "RemoveContainer" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:02.017654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017633    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"} err="failed to get container status \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": rpc error: code = NotFound desc = could not find container \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": container with ID starting with 5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1 not found: ID does not exist"
Sep 17 23:09:02.017696 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017657    2580 scope.go:117] "RemoveContainer" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.017967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017948    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"} err="failed to get container status \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": rpc error: code = NotFound desc = could not find container \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": container with ID starting with 913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2 not found: ID does not exist"
Sep 17 23:09:02.018028 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.017968    2580 scope.go:117] "RemoveContainer" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.018224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.018207    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"} err="failed to get container status \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": rpc error: code = NotFound desc = could not find container \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": container with ID starting with 74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74 not found: ID does not exist"
Sep 17 23:09:02.018266 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.018225    2580 scope.go:117] "RemoveContainer" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:02.018460 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.018444    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"} err="failed to get container status \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": rpc error: code = NotFound desc = could not find container \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": container with ID starting with 5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460 not found: ID does not exist"
Sep 17 23:09:02.018460 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.018460    2580 scope.go:117] "RemoveContainer" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:02.018687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.018667    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"} err="failed to get container status \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": rpc error: code = NotFound desc = could not find container \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": container with ID starting with c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26 not found: ID does not exist"
Sep 17 23:09:02.018755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.018690    2580 scope.go:117] "RemoveContainer" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:02.019035 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.018972    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:09:02.019035 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:09:02.019250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019058    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"} err="failed to get container status \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": rpc error: code = NotFound desc = could not find container \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": container with ID starting with f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761 not found: ID does not exist"
Sep 17 23:09:02.019250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019086    2580 scope.go:117] "RemoveContainer" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:02.019250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:09:02.019250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019039    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:09:02.019250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:09:02.019584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019384    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:09:02.019584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019385    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"} err="failed to get container status \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": rpc error: code = NotFound desc = could not find container \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": container with ID starting with 821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944 not found: ID does not exist"
Sep 17 23:09:02.019584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019404    2580 scope.go:117] "RemoveContainer" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:02.019584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019388    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:09:02.019584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019460    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:09:02.019584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:09:02.020233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019652    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:09:02.020233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:09:02.020233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019703    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"} err="failed to get container status \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": rpc error: code = NotFound desc = could not find container \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": container with ID starting with 5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1 not found: ID does not exist"
Sep 17 23:09:02.020233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019727    2580 scope.go:117] "RemoveContainer" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.020233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.019997    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"} err="failed to get container status \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": rpc error: code = NotFound desc = could not find container \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": container with ID starting with 913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2 not found: ID does not exist"
Sep 17 23:09:02.020233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rr5cr29js559\""
Sep 17 23:09:02.020233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020019    2580 scope.go:117] "RemoveContainer" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.020699 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020247    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"} err="failed to get container status \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": rpc error: code = NotFound desc = could not find container \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": container with ID starting with 74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74 not found: ID does not exist"
Sep 17 23:09:02.020699 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020268    2580 scope.go:117] "RemoveContainer" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:02.020699 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020502    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:09:02.020699 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020517    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5w8gr\""
Sep 17 23:09:02.020699 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020683    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"} err="failed to get container status \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": rpc error: code = NotFound desc = could not find container \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": container with ID starting with 5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460 not found: ID does not exist"
Sep 17 23:09:02.021052 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020706    2580 scope.go:117] "RemoveContainer" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:02.021052 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020960    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"} err="failed to get container status \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": rpc error: code = NotFound desc = could not find container \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": container with ID starting with c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26 not found: ID does not exist"
Sep 17 23:09:02.021052 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.020980    2580 scope.go:117] "RemoveContainer" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:02.021263 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.021238    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"} err="failed to get container status \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": rpc error: code = NotFound desc = could not find container \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": container with ID starting with f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761 not found: ID does not exist"
Sep 17 23:09:02.021367 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.021265    2580 scope.go:117] "RemoveContainer" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:02.022252 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.021916    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"} err="failed to get container status \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": rpc error: code = NotFound desc = could not find container \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": container with ID starting with 821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944 not found: ID does not exist"
Sep 17 23:09:02.022252 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.022116    2580 scope.go:117] "RemoveContainer" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:02.025736 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.025633    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"} err="failed to get container status \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": rpc error: code = NotFound desc = could not find container \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": container with ID starting with 5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1 not found: ID does not exist"
Sep 17 23:09:02.025736 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.025663    2580 scope.go:117] "RemoveContainer" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.026476 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.026452    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"} err="failed to get container status \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": rpc error: code = NotFound desc = could not find container \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": container with ID starting with 913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2 not found: ID does not exist"
Sep 17 23:09:02.026581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.026477    2580 scope.go:117] "RemoveContainer" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.027006 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.026975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"} err="failed to get container status \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": rpc error: code = NotFound desc = could not find container \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": container with ID starting with 74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74 not found: ID does not exist"
Sep 17 23:09:02.027006 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.027004    2580 scope.go:117] "RemoveContainer" containerID="5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"
Sep 17 23:09:02.027711 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.027545    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460"} err="failed to get container status \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": rpc error: code = NotFound desc = could not find container \"5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460\": container with ID starting with 5fb84292d3610db6b9ca9372c757a51739e5ffe65f4859df864006022b2a2460 not found: ID does not exist"
Sep 17 23:09:02.027711 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.027572    2580 scope.go:117] "RemoveContainer" containerID="c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"
Sep 17 23:09:02.027967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.027940    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26"} err="failed to get container status \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": rpc error: code = NotFound desc = could not find container \"c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26\": container with ID starting with c65708f1c07eb86579d0b89568c27be64c0d7b9d015ec19060c58d453067df26 not found: ID does not exist"
Sep 17 23:09:02.028027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.027970    2580 scope.go:117] "RemoveContainer" containerID="f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"
Sep 17 23:09:02.028677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.028638    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761"} err="failed to get container status \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": rpc error: code = NotFound desc = could not find container \"f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761\": container with ID starting with f5aa6f613186e9d020fb946eb83814d877e3d000703bc80f2e00767348587761 not found: ID does not exist"
Sep 17 23:09:02.028677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.028676    2580 scope.go:117] "RemoveContainer" containerID="821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"
Sep 17 23:09:02.028817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.028726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:09:02.028989 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.028968    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944"} err="failed to get container status \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": rpc error: code = NotFound desc = could not find container \"821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944\": container with ID starting with 821fa2620f1dd4c5b1bb825b337055732685ac8dc0d8f635e9f1d606ef08d944 not found: ID does not exist"
Sep 17 23:09:02.029041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.028990    2580 scope.go:117] "RemoveContainer" containerID="5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"
Sep 17 23:09:02.029194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.029173    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:09:02.029251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.029212    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1"} err="failed to get container status \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": rpc error: code = NotFound desc = could not find container \"5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1\": container with ID starting with 5ff4f6da50d57808a8f1591272d99c04a31237299d56649d47854f7e715409d1 not found: ID does not exist"
Sep 17 23:09:02.029251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.029230    2580 scope.go:117] "RemoveContainer" containerID="913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"
Sep 17 23:09:02.029507 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.029486    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2"} err="failed to get container status \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": rpc error: code = NotFound desc = could not find container \"913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2\": container with ID starting with 913602853dae840c3a98e69e1f546c9ca8b1fd8168ef7838cad75114ecd430a2 not found: ID does not exist"
Sep 17 23:09:02.029507 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.029506    2580 scope.go:117] "RemoveContainer" containerID="74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"
Sep 17 23:09:02.029775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.029753    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74"} err="failed to get container status \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": rpc error: code = NotFound desc = could not find container \"74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74\": container with ID starting with 74b5f42beeb9cdc99bcf67be54d522da0b1e6a47d40a9f5e20897b8cccb27e74 not found: ID does not exist"
Sep 17 23:09:02.032614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.032593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:09:02.106991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.106961    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1846824-0aca-42ae-b176-1da0d260c040" path="/var/lib/kubelet/pods/b1846824-0aca-42ae-b176-1da0d260c040/volumes"
Sep 17 23:09:02.145134 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145322 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145322 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145322 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbmw\" (UniqueName: \"kubernetes.io/projected/25013661-efb7-4580-9b68-e00d37f0f5e4-kube-api-access-zwbmw\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-config\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145494 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25013661-efb7-4580-9b68-e00d37f0f5e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145494 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145494 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25013661-efb7-4580-9b68-e00d37f0f5e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145717 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.145717 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.145615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.245950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25013661-efb7-4580-9b68-e00d37f0f5e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246220 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246220 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246220 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbmw\" (UniqueName: \"kubernetes.io/projected/25013661-efb7-4580-9b68-e00d37f0f5e4-kube-api-access-zwbmw\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-config\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25013661-efb7-4580-9b68-e00d37f0f5e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.246972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.246909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.247418 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.247401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.247894 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.247873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.249216 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.249188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.249969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.249375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.249969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.249481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.249969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.249637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.249969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.249644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.249969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.249675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25013661-efb7-4580-9b68-e00d37f0f5e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.251388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.250665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.251388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.250704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.251388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.250768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.251388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.251164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.251388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.251257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.251388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.251343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25013661-efb7-4580-9b68-e00d37f0f5e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.251751 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.251608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.252214 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.252194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-config\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.252294 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.252264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25013661-efb7-4580-9b68-e00d37f0f5e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.253067 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.253052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25013661-efb7-4580-9b68-e00d37f0f5e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.257890 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.257870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbmw\" (UniqueName: \"kubernetes.io/projected/25013661-efb7-4580-9b68-e00d37f0f5e4-kube-api-access-zwbmw\") pod \"prometheus-k8s-0\" (UID: \"25013661-efb7-4580-9b68-e00d37f0f5e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.330051 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.330021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:02.615591 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.615569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:09:02.617132 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:09:02.617103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25013661_efb7_4580_9b68_e00d37f0f5e4.slice/crio-15503bc8b78d4baeb9423ef79b5ce9a38567ef45aa89873a295e8109d1aedbb4 WatchSource:0}: Error finding container 15503bc8b78d4baeb9423ef79b5ce9a38567ef45aa89873a295e8109d1aedbb4: Status 404 returned error can't find the container with id 15503bc8b78d4baeb9423ef79b5ce9a38567ef45aa89873a295e8109d1aedbb4
Sep 17 23:09:02.960925 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.960892    2580 generic.go:356] "Generic (PLEG): container finished" podID="25013661-efb7-4580-9b68-e00d37f0f5e4" containerID="a9c6fcc6247fc8d0f5c730f82fbd906b77c6e6ace1172e6337595ec5abe35aec" exitCode=0
Sep 17 23:09:02.961097 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.960985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerDied","Data":"a9c6fcc6247fc8d0f5c730f82fbd906b77c6e6ace1172e6337595ec5abe35aec"}
Sep 17 23:09:02.961097 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:02.961067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerStarted","Data":"15503bc8b78d4baeb9423ef79b5ce9a38567ef45aa89873a295e8109d1aedbb4"}
Sep 17 23:09:03.969504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:03.969468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerStarted","Data":"cfd720ac5e4b1f7ecfbd8c1a5f0ba6b5d1914c8e125fd8c563efccc7e52db306"}
Sep 17 23:09:03.969889 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:03.969562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerStarted","Data":"dadf8f303559994e18fa39084db84705dea76f305a82b005794ec07d817c32ed"}
Sep 17 23:09:03.969889 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:03.969588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerStarted","Data":"5cd608f9b16d2a47ef5ddba90926587af1ebe775ecb09beadf1bc4e5a2c32149"}
Sep 17 23:09:03.969889 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:03.969612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerStarted","Data":"b8c72532e0bda1ef3717ead82c790815bab6de1b5443c446d10456847818225b"}
Sep 17 23:09:03.969889 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:03.969634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerStarted","Data":"6d3b9ab52a38f9e38804d28a61fc4ae6cc5fd0fd96b6a6ca9a3fb4df038590d3"}
Sep 17 23:09:03.969889 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:03.969657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25013661-efb7-4580-9b68-e00d37f0f5e4","Type":"ContainerStarted","Data":"59449bfd35cc42d03bee21064c6d3010fce11028c2931cbc976e9061eef52607"}
Sep 17 23:09:03.998934 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:03.998876    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.978708733 podStartE2EDuration="2.998856062s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.147329ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:09:03.997181434 +0000 UTC m=+154.427041252" watchObservedRunningTime="2026-09-17 23:09:03.998856062 +0000 UTC m=+154.428715881"
Sep 17 23:09:04.825976 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:04.825897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:04.881374 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:04.881340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:07.330874 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:07.330834    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:12.104559 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:12.104521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:12.204513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:12.204440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:13.927599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:13.927563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:14.055203 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:14.055167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:14.108937 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:14.108905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:14.671242 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:14.671210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:14.732298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:14.732269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:24.970035 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:24.969956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:25.023128 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:25.023083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:25.074991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:25.074966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:33.079289 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:33.079253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:41.249578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:41.249542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:41.301890 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:41.301855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:48.754746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:48.754704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:48.851565 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:09:48.851537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:02.331018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:02.330967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:10:02.347245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:02.347216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:10:03.194929 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:03.194897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:10:10.187658 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:10.187628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:11.579324 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:11.579293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:13.796418 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:13.796385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:13.901244 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:13.901205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:15.639671 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:15.639635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:15.736143 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:15.736113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:16.559567 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:16.559536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:16.688938 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:16.688908    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:17.678184 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:17.678153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:18.047224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.047141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:18.444261 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.444230    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"]
Sep 17 23:10:18.447568 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.447551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:18.450059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.450035    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 23:10:18.450059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.450053    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 23:10:18.450918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.450903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9fmrl\""
Sep 17 23:10:18.457062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.457043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"]
Sep 17 23:10:18.557473 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.557444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75n9\" (UniqueName: \"kubernetes.io/projected/4eced6e9-0136-4e33-83f9-7285091a6241-kube-api-access-f75n9\") pod \"cert-manager-cainjector-559cdd8c87-4nd8d\" (UID: \"4eced6e9-0136-4e33-83f9-7285091a6241\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:18.557633 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.557507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4eced6e9-0136-4e33-83f9-7285091a6241-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4nd8d\" (UID: \"4eced6e9-0136-4e33-83f9-7285091a6241\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:18.657908 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.657879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4eced6e9-0136-4e33-83f9-7285091a6241-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4nd8d\" (UID: \"4eced6e9-0136-4e33-83f9-7285091a6241\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:18.658065 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.657928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75n9\" (UniqueName: \"kubernetes.io/projected/4eced6e9-0136-4e33-83f9-7285091a6241-kube-api-access-f75n9\") pod \"cert-manager-cainjector-559cdd8c87-4nd8d\" (UID: \"4eced6e9-0136-4e33-83f9-7285091a6241\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:18.666104 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.666076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4eced6e9-0136-4e33-83f9-7285091a6241-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-4nd8d\" (UID: \"4eced6e9-0136-4e33-83f9-7285091a6241\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:18.666235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.666145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75n9\" (UniqueName: \"kubernetes.io/projected/4eced6e9-0136-4e33-83f9-7285091a6241-kube-api-access-f75n9\") pod \"cert-manager-cainjector-559cdd8c87-4nd8d\" (UID: \"4eced6e9-0136-4e33-83f9-7285091a6241\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:18.763321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:18.763233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"
Sep 17 23:10:19.032229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:19.032148    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d"]
Sep 17 23:10:19.035227 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:10:19.035199    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eced6e9_0136_4e33_83f9_7285091a6241.slice/crio-ecbf2fbdeeff9b77377dd85600e8c9e991514b476bdc800968e0619fb690ab46 WatchSource:0}: Error finding container ecbf2fbdeeff9b77377dd85600e8c9e991514b476bdc800968e0619fb690ab46: Status 404 returned error can't find the container with id ecbf2fbdeeff9b77377dd85600e8c9e991514b476bdc800968e0619fb690ab46
Sep 17 23:10:19.238678 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:19.238644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d" event={"ID":"4eced6e9-0136-4e33-83f9-7285091a6241","Type":"ContainerStarted","Data":"ecbf2fbdeeff9b77377dd85600e8c9e991514b476bdc800968e0619fb690ab46"}
Sep 17 23:10:20.998767 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:20.998728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:21.093729 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:21.093690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:25.715519 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:25.715488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:30.283032 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:30.282935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d" event={"ID":"4eced6e9-0136-4e33-83f9-7285091a6241","Type":"ContainerStarted","Data":"87a36216df527753dbdcbd4c113119a33a1900e5e6e2d87554c26b97fb93a8ca"}
Sep 17 23:10:30.299916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:30.299854    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-4nd8d" podStartSLOduration=1.41649215 podStartE2EDuration="12.299832154s" totalImagesPullingTime="10.883340004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:30.297113158 +0000 UTC m=+240.726972976" watchObservedRunningTime="2026-09-17 23:10:30.299832154 +0000 UTC m=+240.729691991"
Sep 17 23:10:30.684755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:30.684723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:30.784775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:30.784739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:32.654523 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:32.654464    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:32.761235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:32.761197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:34.641279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.641244    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"]
Sep 17 23:10:34.644372 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.644357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:34.646913 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.646894    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:34.646913 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.646904    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-r2t7h\""
Sep 17 23:10:34.647069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.646897    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:34.655404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.655385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"]
Sep 17 23:10:34.800498 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.800463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jxs\" (UniqueName: \"kubernetes.io/projected/de8420c2-239b-4dcf-9b19-7ff5b48fe380-kube-api-access-c6jxs\") pod \"openshift-lws-operator-fd8ccff4c-jdfjf\" (UID: \"de8420c2-239b-4dcf-9b19-7ff5b48fe380\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:34.800681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.800529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de8420c2-239b-4dcf-9b19-7ff5b48fe380-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jdfjf\" (UID: \"de8420c2-239b-4dcf-9b19-7ff5b48fe380\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:34.901644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.901568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jxs\" (UniqueName: \"kubernetes.io/projected/de8420c2-239b-4dcf-9b19-7ff5b48fe380-kube-api-access-c6jxs\") pod \"openshift-lws-operator-fd8ccff4c-jdfjf\" (UID: \"de8420c2-239b-4dcf-9b19-7ff5b48fe380\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:34.901796 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.901662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de8420c2-239b-4dcf-9b19-7ff5b48fe380-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jdfjf\" (UID: \"de8420c2-239b-4dcf-9b19-7ff5b48fe380\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:34.902092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.902074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de8420c2-239b-4dcf-9b19-7ff5b48fe380-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jdfjf\" (UID: \"de8420c2-239b-4dcf-9b19-7ff5b48fe380\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:34.909880 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.909858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jxs\" (UniqueName: \"kubernetes.io/projected/de8420c2-239b-4dcf-9b19-7ff5b48fe380-kube-api-access-c6jxs\") pod \"openshift-lws-operator-fd8ccff4c-jdfjf\" (UID: \"de8420c2-239b-4dcf-9b19-7ff5b48fe380\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:34.953341 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:34.953308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"
Sep 17 23:10:35.224500 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:35.224428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf"]
Sep 17 23:10:35.227479 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:10:35.227436    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde8420c2_239b_4dcf_9b19_7ff5b48fe380.slice/crio-6a495e165d7367f78282dcc4c69c35fcad4a4a4bb37ece6434e10d4fde16945d WatchSource:0}: Error finding container 6a495e165d7367f78282dcc4c69c35fcad4a4a4bb37ece6434e10d4fde16945d: Status 404 returned error can't find the container with id 6a495e165d7367f78282dcc4c69c35fcad4a4a4bb37ece6434e10d4fde16945d
Sep 17 23:10:35.302503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:35.302457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf" event={"ID":"de8420c2-239b-4dcf-9b19-7ff5b48fe380","Type":"ContainerStarted","Data":"6a495e165d7367f78282dcc4c69c35fcad4a4a4bb37ece6434e10d4fde16945d"}
Sep 17 23:10:38.315664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:38.315618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf" event={"ID":"de8420c2-239b-4dcf-9b19-7ff5b48fe380","Type":"ContainerStarted","Data":"45b5f17f5ac3a0d18b252db29c048d1f2a3c739c41720c0a28d7118130da528b"}
Sep 17 23:10:38.332338 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:38.332282    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jdfjf" podStartSLOduration=1.7595142849999998 podStartE2EDuration="4.332256041s" totalImagesPullingTime="2.572741756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:38.331655631 +0000 UTC m=+248.761515448" watchObservedRunningTime="2026-09-17 23:10:38.332256041 +0000 UTC m=+248.762115860"
Sep 17 23:10:38.974930 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:38.974895    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:39.074921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:39.074871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:40.728953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:40.728924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:45.143149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:45.143114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:46.064721 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:46.064687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:55.022462 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:55.022371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:55.130690 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:55.130661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:55.625946 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:55.625913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:55.906825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:55.906772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.003900 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.003868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.054776 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.054739    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"]
Sep 17 23:10:56.061591 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.061567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.064286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.064266    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 23:10:56.064413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.064265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 23:10:56.064763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.064745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 23:10:56.064995 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.064979    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lj7s7\""
Sep 17 23:10:56.065050 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.064982    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 23:10:56.076821 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.076775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"]
Sep 17 23:10:56.078486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.078466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56vg\" (UniqueName: \"kubernetes.io/projected/890439e4-9ff6-4ade-983d-8064e83e606a-kube-api-access-h56vg\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.078573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.078492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890439e4-9ff6-4ade-983d-8064e83e606a-tmp\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.078573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.078520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/890439e4-9ff6-4ade-983d-8064e83e606a-manifests\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.078573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.078547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/890439e4-9ff6-4ade-983d-8064e83e606a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.078683 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.078581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/890439e4-9ff6-4ade-983d-8064e83e606a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.179557 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.179460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56vg\" (UniqueName: \"kubernetes.io/projected/890439e4-9ff6-4ade-983d-8064e83e606a-kube-api-access-h56vg\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.179557 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.179519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890439e4-9ff6-4ade-983d-8064e83e606a-tmp\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.179754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.179561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/890439e4-9ff6-4ade-983d-8064e83e606a-manifests\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.179754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.179595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/890439e4-9ff6-4ade-983d-8064e83e606a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.179754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.179628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/890439e4-9ff6-4ade-983d-8064e83e606a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.180046 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.180020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/890439e4-9ff6-4ade-983d-8064e83e606a-manifests\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.182140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.182111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/890439e4-9ff6-4ade-983d-8064e83e606a-tmp\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.182250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.182199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/890439e4-9ff6-4ade-983d-8064e83e606a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.182314 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.182258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/890439e4-9ff6-4ade-983d-8064e83e606a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.192099 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.192064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56vg\" (UniqueName: \"kubernetes.io/projected/890439e4-9ff6-4ade-983d-8064e83e606a-kube-api-access-h56vg\") pod \"opendatahub-operator-controller-manager-5d5976b594-fjzqw\" (UID: \"890439e4-9ff6-4ade-983d-8064e83e606a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.253745 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.253711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.354139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.354096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.371562 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.371537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:10:56.454072 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.453978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.555204 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.555167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.656472 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.656433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"]
Sep 17 23:10:56.659488 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:56.659424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.659753 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:10:56.659731    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890439e4_9ff6_4ade_983d_8064e83e606a.slice/crio-197b69857a30e39c6c58036aad73401d592203aa0b79389c182ae9f780f63592 WatchSource:0}: Error finding container 197b69857a30e39c6c58036aad73401d592203aa0b79389c182ae9f780f63592: Status 404 returned error can't find the container with id 197b69857a30e39c6c58036aad73401d592203aa0b79389c182ae9f780f63592
Sep 17 23:10:57.397355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:10:57.397312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw" event={"ID":"890439e4-9ff6-4ade-983d-8064e83e606a","Type":"ContainerStarted","Data":"197b69857a30e39c6c58036aad73401d592203aa0b79389c182ae9f780f63592"}
Sep 17 23:11:00.412005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:00.411966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw" event={"ID":"890439e4-9ff6-4ade-983d-8064e83e606a","Type":"ContainerStarted","Data":"1dfcee6cc1212e0645901ad44055300225969089c1a0be64df2d1c8d39468f8b"}
Sep 17 23:11:01.028026 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:01.027993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:01.121668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:01.121621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:01.418276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:01.418245    2580 generic.go:356] "Generic (PLEG): container finished" podID="890439e4-9ff6-4ade-983d-8064e83e606a" containerID="1dfcee6cc1212e0645901ad44055300225969089c1a0be64df2d1c8d39468f8b" exitCode=0
Sep 17 23:11:01.418676 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:01.418297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw" event={"ID":"890439e4-9ff6-4ade-983d-8064e83e606a","Type":"ContainerDied","Data":"1dfcee6cc1212e0645901ad44055300225969089c1a0be64df2d1c8d39468f8b"}
Sep 17 23:11:02.425029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:02.424995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw" event={"ID":"890439e4-9ff6-4ade-983d-8064e83e606a","Type":"ContainerStarted","Data":"b3746a987b8ba1da009757526340a29eee1cd46b36944c85a87ef6fdce646afc"}
Sep 17 23:11:02.425474 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:02.425070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:11:02.448325 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:02.448274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw" podStartSLOduration=1.537208575 podStartE2EDuration="6.448255839s" totalImagesPullingTime="3.849933846s" totalInitContainerRuntime="1.061113418s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.447379767 +0000 UTC m=+272.877239586" watchObservedRunningTime="2026-09-17 23:11:02.448255839 +0000 UTC m=+272.878115657"
Sep 17 23:11:03.258267 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:03.258236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:03.312331 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:03.312298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:13.428497 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.428463    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"]
Sep 17 23:11:13.436818 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.436775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5976b594-fjzqw"
Sep 17 23:11:13.436956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.436894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.439286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.439251    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 23:11:13.439947 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.439922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 23:11:13.439947 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.439937    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 23:11:13.440132 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.440014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zfkq4\""
Sep 17 23:11:13.440132 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.440055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"]
Sep 17 23:11:13.523436 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.523407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-metrics-cert\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.523617 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.523458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4pf\" (UniqueName: \"kubernetes.io/projected/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-kube-api-access-5s4pf\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.523617 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.523502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-manager-config\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.523617 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.523550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-cert\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.624373 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.624326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4pf\" (UniqueName: \"kubernetes.io/projected/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-kube-api-access-5s4pf\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.624545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.624378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-manager-config\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.624545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.624442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-cert\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.624622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.624584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-metrics-cert\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.625133 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.625111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-manager-config\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.627124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.627103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-metrics-cert\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.627223 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.627192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-cert\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.633177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.633154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4pf\" (UniqueName: \"kubernetes.io/projected/0b34226a-924d-4cb7-a0d4-ada2a4a5b79a-kube-api-access-5s4pf\") pod \"lws-controller-manager-ccddb67c-xtbq4\" (UID: \"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a\") " pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:13.747954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:13.747861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:14.028686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:14.028612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"]
Sep 17 23:11:14.031670 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:11:14.031633    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b34226a_924d_4cb7_a0d4_ada2a4a5b79a.slice/crio-39950aab0e64eff122e06965a5a262fe5fdaaecd134975478e77c915e9a2d76d WatchSource:0}: Error finding container 39950aab0e64eff122e06965a5a262fe5fdaaecd134975478e77c915e9a2d76d: Status 404 returned error can't find the container with id 39950aab0e64eff122e06965a5a262fe5fdaaecd134975478e77c915e9a2d76d
Sep 17 23:11:14.474434 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:14.474400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4" event={"ID":"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a","Type":"ContainerStarted","Data":"39950aab0e64eff122e06965a5a262fe5fdaaecd134975478e77c915e9a2d76d"}
Sep 17 23:11:14.980999 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:14.980931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.146871 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.146831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.465764 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.465651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.476794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.476763    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"]
Sep 17 23:11:15.480261 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.480236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.481574 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.481546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4" event={"ID":"0b34226a-924d-4cb7-a0d4-ada2a4a5b79a","Type":"ContainerStarted","Data":"d48fc408e146e7e1b9131f51f3f2570b069857f45192a9f6e815cdf2d924138b"}
Sep 17 23:11:15.481697 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.481680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:15.482624 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.482572    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 23:11:15.482742 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.482629    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 23:11:15.482742 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.482737    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dvdfp\""
Sep 17 23:11:15.482890 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.482747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 23:11:15.483195 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.483182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 23:11:15.502054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.502021    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"]
Sep 17 23:11:15.539920 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.539887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-tmp\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.540128 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.540031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxwx\" (UniqueName: \"kubernetes.io/projected/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-kube-api-access-6jxwx\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.540128 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.540093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-tls-certs\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.555817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.555737    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4" podStartSLOduration=1.25124056 podStartE2EDuration="2.555718209s" totalImagesPullingTime="1.304477649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:15.551919893 +0000 UTC m=+285.981779711" watchObservedRunningTime="2026-09-17 23:11:15.555718209 +0000 UTC m=+285.985578026"
Sep 17 23:11:15.641400 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.641374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxwx\" (UniqueName: \"kubernetes.io/projected/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-kube-api-access-6jxwx\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.641605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.641416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-tls-certs\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.641605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.641438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-tmp\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.644168 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.644132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-tmp\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.644346 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.644325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-tls-certs\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.652235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.652205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxwx\" (UniqueName: \"kubernetes.io/projected/80dc2e8e-b2de-4d56-bed1-1b6f20e6021a-kube-api-access-6jxwx\") pod \"kube-auth-proxy-55b8d65647-7mdqg\" (UID: \"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a\") " pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:15.673773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.673739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:15.790475 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:15.790436    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"
Sep 17 23:11:16.083966 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:16.083941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg"]
Sep 17 23:11:16.085241 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:11:16.085211    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dc2e8e_b2de_4d56_bed1_1b6f20e6021a.slice/crio-a1b0cc8411906f27da2083091e776b920c44bedc816aaddd86e5267ddb7d79b8 WatchSource:0}: Error finding container a1b0cc8411906f27da2083091e776b920c44bedc816aaddd86e5267ddb7d79b8: Status 404 returned error can't find the container with id a1b0cc8411906f27da2083091e776b920c44bedc816aaddd86e5267ddb7d79b8
Sep 17 23:11:16.487512 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:16.487463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg" event={"ID":"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a","Type":"ContainerStarted","Data":"a1b0cc8411906f27da2083091e776b920c44bedc816aaddd86e5267ddb7d79b8"}
Sep 17 23:11:17.173003 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:17.172960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:17.302125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:17.302039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:18.243284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:18.243246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:19.504486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:19.504444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg" event={"ID":"80dc2e8e-b2de-4d56-bed1-1b6f20e6021a","Type":"ContainerStarted","Data":"fe499dd9740c034617beedd4efc4ee23707e4f81ff35ba8d544201a114bb9b15"}
Sep 17 23:11:19.521774 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:19.521722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55b8d65647-7mdqg" podStartSLOduration=1.680319143 podStartE2EDuration="4.521703378s" totalImagesPullingTime="2.841384235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:19.520466036 +0000 UTC m=+289.950325853" watchObservedRunningTime="2026-09-17 23:11:19.521703378 +0000 UTC m=+289.951563196"
Sep 17 23:11:22.709729 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:22.709693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:22.807733 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:22.807696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:23.009609 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:23.009520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:23.112420 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:23.112387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:23.269082 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:23.269007    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:26.037121 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:26.037078    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:26.489850 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:26.489791    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ccddb67c-xtbq4"
Sep 17 23:11:27.123055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:27.123022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:28.868735 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:28.868701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:28.969621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:28.969590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:29.995201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:29.995171    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:11:29.998746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:29.998719    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:11:29.998917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:29.998901    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:11:35.686002 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:35.685972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.797653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:35.797621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.902756 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:35.902720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:36.006537 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:36.006446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:37.016500 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:37.016468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:37.116412 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:37.116383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:37.217219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:37.217186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:57.113949 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:57.113906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:57.177571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:11:57.177535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.052546 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:27.052441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.151612 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:27.151569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:28.912702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.912668    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-644f54f8c5-57q6p"]
Sep 17 23:12:28.915993 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.915975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:28.919198 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919160    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t72j2\""
Sep 17 23:12:28.919371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919160    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:12:28.919371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:12:28.919371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:12:28.919371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919254    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:12:28.919668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919652    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:12:28.919716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:12:28.919768 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.919753    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:12:28.924863 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.924846    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 23:12:28.928692 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.928668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644f54f8c5-57q6p"]
Sep 17 23:12:28.975166 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:28.975137    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:29.051149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.051112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-oauth-config\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.051149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.051150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-serving-cert\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.051342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.051209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-service-ca\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.051342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.051259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqgh\" (UniqueName: \"kubernetes.io/projected/66385567-6f11-43b7-b2a4-a112aa8edd7f-kube-api-access-fpqgh\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.051342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.051300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-trusted-ca-bundle\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.051342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.051331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-config\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.051469 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.051345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-oauth-serving-cert\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.076179 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.076150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:29.152218 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.152184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-config\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.152218 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.152215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-oauth-serving-cert\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.152420 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.152249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-oauth-config\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.152420 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.152267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-serving-cert\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.152420 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.152316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-service-ca\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.152420 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.152342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqgh\" (UniqueName: \"kubernetes.io/projected/66385567-6f11-43b7-b2a4-a112aa8edd7f-kube-api-access-fpqgh\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.152620 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.152596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-trusted-ca-bundle\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.153374 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.153344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-oauth-serving-cert\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.153507 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.153394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-service-ca\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.153990 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.153967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-config\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.155871 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.155841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66385567-6f11-43b7-b2a4-a112aa8edd7f-trusted-ca-bundle\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.156245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.156220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-serving-cert\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.159915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.159887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66385567-6f11-43b7-b2a4-a112aa8edd7f-console-oauth-config\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.161541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.161524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqgh\" (UniqueName: \"kubernetes.io/projected/66385567-6f11-43b7-b2a4-a112aa8edd7f-kube-api-access-fpqgh\") pod \"console-644f54f8c5-57q6p\" (UID: \"66385567-6f11-43b7-b2a4-a112aa8edd7f\") " pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.176099 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.176035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:29.226351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.226314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:29.497761 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.497737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644f54f8c5-57q6p"]
Sep 17 23:12:29.500103 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:12:29.500068    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66385567_6f11_43b7_b2a4_a112aa8edd7f.slice/crio-6c54fb40d353b41bff4cfe319ef91d49c7cf7b6cd6c38de1d3db7d3ed708cf1c WatchSource:0}: Error finding container 6c54fb40d353b41bff4cfe319ef91d49c7cf7b6cd6c38de1d3db7d3ed708cf1c: Status 404 returned error can't find the container with id 6c54fb40d353b41bff4cfe319ef91d49c7cf7b6cd6c38de1d3db7d3ed708cf1c
Sep 17 23:12:29.674962 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.674928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:29.799078 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.798978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644f54f8c5-57q6p" event={"ID":"66385567-6f11-43b7-b2a4-a112aa8edd7f","Type":"ContainerStarted","Data":"e24b0aca125dc2b169386ed621a235db2fed4b9e7f79b6ef6dda68e3fa0aa614"}
Sep 17 23:12:29.799078 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.799039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644f54f8c5-57q6p" event={"ID":"66385567-6f11-43b7-b2a4-a112aa8edd7f","Type":"ContainerStarted","Data":"6c54fb40d353b41bff4cfe319ef91d49c7cf7b6cd6c38de1d3db7d3ed708cf1c"}
Sep 17 23:12:29.816935 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:29.816878    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-644f54f8c5-57q6p" podStartSLOduration=1.816859602 podStartE2EDuration="1.816859602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:29.816123716 +0000 UTC m=+360.245983543" watchObservedRunningTime="2026-09-17 23:12:29.816859602 +0000 UTC m=+360.246719420"
Sep 17 23:12:38.255216 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:38.255185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:38.308793 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:38.308760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:39.227020 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:39.226978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:39.227217 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:39.227049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:39.231720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:39.231699    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:39.843352 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:39.843320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-644f54f8c5-57q6p"
Sep 17 23:12:40.021556 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:40.021520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:40.122258 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:40.122156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:40.223320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:40.223291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:41.333942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:41.333903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:43.933058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:43.933023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:44.333139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:44.333047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:44.733883 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:44.733853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:47.634654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:47.634581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:47.736479 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:47.736445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:48.331865 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:48.331830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:48.612789 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:48.612698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:48.720301 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:48.720265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.088868 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:50.088830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.190817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:50.190769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.948376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:50.948344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:51.568486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:51.568455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:52.616308 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.616257    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"]
Sep 17 23:12:52.619502 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.619486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"
Sep 17 23:12:52.622170 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.622152    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 23:12:52.622313 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.622153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fv9vt\""
Sep 17 23:12:52.622492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.622475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 23:12:52.622847 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.622833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 23:12:52.633957 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.633933    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"]
Sep 17 23:12:52.659107 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.659075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vzg\" (UniqueName: \"kubernetes.io/projected/e7a87d72-1d6a-4278-a0c2-51f3b35e809a-kube-api-access-w4vzg\") pod \"dns-operator-controller-manager-7b84cd7c9c-g6rsg\" (UID: \"e7a87d72-1d6a-4278-a0c2-51f3b35e809a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"
Sep 17 23:12:52.759920 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.759887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vzg\" (UniqueName: \"kubernetes.io/projected/e7a87d72-1d6a-4278-a0c2-51f3b35e809a-kube-api-access-w4vzg\") pod \"dns-operator-controller-manager-7b84cd7c9c-g6rsg\" (UID: \"e7a87d72-1d6a-4278-a0c2-51f3b35e809a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"
Sep 17 23:12:52.773160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.773132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vzg\" (UniqueName: \"kubernetes.io/projected/e7a87d72-1d6a-4278-a0c2-51f3b35e809a-kube-api-access-w4vzg\") pod \"dns-operator-controller-manager-7b84cd7c9c-g6rsg\" (UID: \"e7a87d72-1d6a-4278-a0c2-51f3b35e809a\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"
Sep 17 23:12:52.929882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:52.929838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"
Sep 17 23:12:53.210688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:53.210611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"]
Sep 17 23:12:53.213543 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:12:53.213512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a87d72_1d6a_4278_a0c2_51f3b35e809a.slice/crio-6a9dc846374b93ddcc0263d88858e7552eac01ebf7262d9e437c1a65d2c5c78f WatchSource:0}: Error finding container 6a9dc846374b93ddcc0263d88858e7552eac01ebf7262d9e437c1a65d2c5c78f: Status 404 returned error can't find the container with id 6a9dc846374b93ddcc0263d88858e7552eac01ebf7262d9e437c1a65d2c5c78f
Sep 17 23:12:53.215662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:53.215644    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:12:53.902491 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:53.902450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg" event={"ID":"e7a87d72-1d6a-4278-a0c2-51f3b35e809a","Type":"ContainerStarted","Data":"6a9dc846374b93ddcc0263d88858e7552eac01ebf7262d9e437c1a65d2c5c78f"}
Sep 17 23:12:53.944539 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:53.944501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.047681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:54.047648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.167614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.167576    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"]
Sep 17 23:12:55.170574 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.170546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.172165 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.172147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.174914 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.174895    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rdj4p\""
Sep 17 23:12:55.175229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.175210    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 23:12:55.175309 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.175245    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 23:12:55.183950 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.183928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"]
Sep 17 23:12:55.278760 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.278713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27571f8a-284d-47fc-9392-2278a95fa8f4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.278979 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.278866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27571f8a-284d-47fc-9392-2278a95fa8f4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.278979 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.278925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2nv\" (UniqueName: \"kubernetes.io/projected/27571f8a-284d-47fc-9392-2278a95fa8f4-kube-api-access-wb2nv\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.329100 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.329063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.379443 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.379408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27571f8a-284d-47fc-9392-2278a95fa8f4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.379790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.379456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2nv\" (UniqueName: \"kubernetes.io/projected/27571f8a-284d-47fc-9392-2278a95fa8f4-kube-api-access-wb2nv\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.379790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.379478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27571f8a-284d-47fc-9392-2278a95fa8f4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.380299 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.380267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27571f8a-284d-47fc-9392-2278a95fa8f4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.382117 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.382089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27571f8a-284d-47fc-9392-2278a95fa8f4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.389634 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.389612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2nv\" (UniqueName: \"kubernetes.io/projected/27571f8a-284d-47fc-9392-2278a95fa8f4-kube-api-access-wb2nv\") pod \"kuadrant-console-plugin-fbdd4f445-lhr6j\" (UID: \"27571f8a-284d-47fc-9392-2278a95fa8f4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.481256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.481179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"
Sep 17 23:12:55.754600 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.754574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j"]
Sep 17 23:12:55.756868 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:12:55.756837    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27571f8a_284d_47fc_9392_2278a95fa8f4.slice/crio-5ae5fa2d7d359d39bbaf28eab6b6f8fc03b642f0c81f94eeb954813f3648c597 WatchSource:0}: Error finding container 5ae5fa2d7d359d39bbaf28eab6b6f8fc03b642f0c81f94eeb954813f3648c597: Status 404 returned error can't find the container with id 5ae5fa2d7d359d39bbaf28eab6b6f8fc03b642f0c81f94eeb954813f3648c597
Sep 17 23:12:55.912371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:55.912336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j" event={"ID":"27571f8a-284d-47fc-9392-2278a95fa8f4","Type":"ContainerStarted","Data":"5ae5fa2d7d359d39bbaf28eab6b6f8fc03b642f0c81f94eeb954813f3648c597"}
Sep 17 23:12:57.937350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:57.937311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.041921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:58.041877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.341841 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:58.341753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.406619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:58.406578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.547694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:58.547659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.629443 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:12:58.629354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.941200 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:00.941158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j" event={"ID":"27571f8a-284d-47fc-9392-2278a95fa8f4","Type":"ContainerStarted","Data":"787a1ca195b2a75e44d4ba9576887db82c8b2d0661f9aaf1dc4966364728ebf6"}
Sep 17 23:13:00.959568 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:00.959518    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-lhr6j" podStartSLOduration=1.353391102 podStartE2EDuration="5.95949635s" totalImagesPullingTime="4.606105248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:00.958426342 +0000 UTC m=+391.388286181" watchObservedRunningTime="2026-09-17 23:13:00.95949635 +0000 UTC m=+391.389356168"
Sep 17 23:13:02.952785 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:02.952749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg" event={"ID":"e7a87d72-1d6a-4278-a0c2-51f3b35e809a","Type":"ContainerStarted","Data":"ae1f683b1926b591ed84723ca029acc7424c3d8c246c52f3b360d9e01b874b01"}
Sep 17 23:13:02.953190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:02.952883    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"
Sep 17 23:13:02.973189 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:02.973134    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg" podStartSLOduration=1.9774548589999998 podStartE2EDuration="10.973116825s" totalImagesPullingTime="8.995661966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:02.971518056 +0000 UTC m=+393.401377880" watchObservedRunningTime="2026-09-17 23:13:02.973116825 +0000 UTC m=+393.402976642"
Sep 17 23:13:07.078293 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.078258    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:07.223416 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.223382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:07.372820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.372720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:07.504149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.504114    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"]
Sep 17 23:13:07.507461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.507444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:07.512196 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.512175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t687b\""
Sep 17 23:13:07.524177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.524149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"]
Sep 17 23:13:07.592505 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.592476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmqz\" (UniqueName: \"kubernetes.io/projected/ad5da990-f291-48f6-91dd-96b54e91d5c3-kube-api-access-tlmqz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kg65f\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:07.592681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.592563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad5da990-f291-48f6-91dd-96b54e91d5c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kg65f\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:07.693433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.693395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad5da990-f291-48f6-91dd-96b54e91d5c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kg65f\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:07.693628 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.693450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmqz\" (UniqueName: \"kubernetes.io/projected/ad5da990-f291-48f6-91dd-96b54e91d5c3-kube-api-access-tlmqz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kg65f\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:07.693868 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.693846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad5da990-f291-48f6-91dd-96b54e91d5c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kg65f\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:07.706785 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.706754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmqz\" (UniqueName: \"kubernetes.io/projected/ad5da990-f291-48f6-91dd-96b54e91d5c3-kube-api-access-tlmqz\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kg65f\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:07.817298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:07.817263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:08.114116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:08.114027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"]
Sep 17 23:13:08.117893 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:13:08.117859    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5da990_f291_48f6_91dd_96b54e91d5c3.slice/crio-9257248d4701bdf1afe2062b7ee44fc16a2813ef8c2853d741c6feb8cd4447dd WatchSource:0}: Error finding container 9257248d4701bdf1afe2062b7ee44fc16a2813ef8c2853d741c6feb8cd4447dd: Status 404 returned error can't find the container with id 9257248d4701bdf1afe2062b7ee44fc16a2813ef8c2853d741c6feb8cd4447dd
Sep 17 23:13:08.983278 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:08.983240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" event={"ID":"ad5da990-f291-48f6-91dd-96b54e91d5c3","Type":"ContainerStarted","Data":"9257248d4701bdf1afe2062b7ee44fc16a2813ef8c2853d741c6feb8cd4447dd"}
Sep 17 23:13:09.418266 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:09.418218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:09.523504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:09.523465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:12.001766 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:12.001656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" event={"ID":"ad5da990-f291-48f6-91dd-96b54e91d5c3","Type":"ContainerStarted","Data":"3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92"}
Sep 17 23:13:12.002245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:12.001833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:12.031898 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:12.031846    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" podStartSLOduration=1.400508692 podStartE2EDuration="5.031831061s" totalImagesPullingTime="3.631322369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:12.030127114 +0000 UTC m=+402.459986934" watchObservedRunningTime="2026-09-17 23:13:12.031831061 +0000 UTC m=+402.461690878"
Sep 17 23:13:13.689686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:13.689651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:13.815914 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:13.815885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:13.910958 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:13.910926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:13.959778 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:13.959705    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g6rsg"
Sep 17 23:13:17.563168 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:17.563134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:17.663211 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:17.663182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:17.764098 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:17.764064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:17.823387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:17.823314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:18.771793 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:18.771761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:18.859716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:18.859680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:23.009090 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:23.009060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:13:26.797428 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:26.797386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:26.896345 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:26.896306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:29.224063 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:29.224025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:29.324634 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:29.324597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:39.491186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:39.491152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:39.673774 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:39.673744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:44.842656 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:44.842617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:44.981056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:44.981018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:45.093739 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.093655    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:45.712264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.712226    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c44x2"]
Sep 17 23:13:45.733079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.733048    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c44x2"]
Sep 17 23:13:45.733239 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.733143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2"
Sep 17 23:13:45.736029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.735993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cgpp7\""
Sep 17 23:13:45.751510 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.751478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:45.824351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.824316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4kv\" (UniqueName: \"kubernetes.io/projected/dfbfe511-148f-437d-b521-dd4091e9a8bb-kube-api-access-jj4kv\") pod \"authorino-7bcfd8f9cd-c44x2\" (UID: \"dfbfe511-148f-437d-b521-dd4091e9a8bb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2"
Sep 17 23:13:45.848699 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.848668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:45.925640 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.925596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4kv\" (UniqueName: \"kubernetes.io/projected/dfbfe511-148f-437d-b521-dd4091e9a8bb-kube-api-access-jj4kv\") pod \"authorino-7bcfd8f9cd-c44x2\" (UID: \"dfbfe511-148f-437d-b521-dd4091e9a8bb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2"
Sep 17 23:13:45.939520 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:45.939497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4kv\" (UniqueName: \"kubernetes.io/projected/dfbfe511-148f-437d-b521-dd4091e9a8bb-kube-api-access-jj4kv\") pod \"authorino-7bcfd8f9cd-c44x2\" (UID: \"dfbfe511-148f-437d-b521-dd4091e9a8bb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2"
Sep 17 23:13:46.043398 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:46.043300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2"
Sep 17 23:13:46.312631 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:46.312553    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c44x2"]
Sep 17 23:13:46.315989 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:13:46.315961    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfbfe511_148f_437d_b521_dd4091e9a8bb.slice/crio-213f3fef5e2cea35c0f67090f995f12570d05e8995e3302aad7dfa285784bd5d WatchSource:0}: Error finding container 213f3fef5e2cea35c0f67090f995f12570d05e8995e3302aad7dfa285784bd5d: Status 404 returned error can't find the container with id 213f3fef5e2cea35c0f67090f995f12570d05e8995e3302aad7dfa285784bd5d
Sep 17 23:13:47.162209 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:47.162168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2" event={"ID":"dfbfe511-148f-437d-b521-dd4091e9a8bb","Type":"ContainerStarted","Data":"213f3fef5e2cea35c0f67090f995f12570d05e8995e3302aad7dfa285784bd5d"}
Sep 17 23:13:50.173068 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:50.173034    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c44x2"]
Sep 17 23:13:54.195376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:54.195331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2" event={"ID":"dfbfe511-148f-437d-b521-dd4091e9a8bb","Type":"ContainerStarted","Data":"db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8"}
Sep 17 23:13:54.195376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:54.195338    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2" podUID="dfbfe511-148f-437d-b521-dd4091e9a8bb" containerName="authorino" containerID="cri-o://db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8" gracePeriod=30
Sep 17 23:13:54.211119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:54.211075    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2" podStartSLOduration=2.258335167 podStartE2EDuration="9.211058415s" totalImagesPullingTime="6.952723248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:54.20952521 +0000 UTC m=+444.639385038" watchObservedRunningTime="2026-09-17 23:13:54.211058415 +0000 UTC m=+444.640918243"
Sep 17 23:13:54.442768 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:54.442740    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2"
Sep 17 23:13:54.502924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:54.502839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj4kv\" (UniqueName: \"kubernetes.io/projected/dfbfe511-148f-437d-b521-dd4091e9a8bb-kube-api-access-jj4kv\") pod \"dfbfe511-148f-437d-b521-dd4091e9a8bb\" (UID: \"dfbfe511-148f-437d-b521-dd4091e9a8bb\") "
Sep 17 23:13:54.505226 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:54.505199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbfe511-148f-437d-b521-dd4091e9a8bb-kube-api-access-jj4kv" (OuterVolumeSpecName: "kube-api-access-jj4kv") pod "dfbfe511-148f-437d-b521-dd4091e9a8bb" (UID: "dfbfe511-148f-437d-b521-dd4091e9a8bb"). InnerVolumeSpecName "kube-api-access-jj4kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:54.603904 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:54.603868    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj4kv\" (UniqueName: \"kubernetes.io/projected/dfbfe511-148f-437d-b521-dd4091e9a8bb-kube-api-access-jj4kv\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:13:55.200969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.200933    2580 generic.go:356] "Generic (PLEG): container finished" podID="dfbfe511-148f-437d-b521-dd4091e9a8bb" containerID="db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8" exitCode=0
Sep 17 23:13:55.201395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.200975    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2"
Sep 17 23:13:55.201395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.200981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2" event={"ID":"dfbfe511-148f-437d-b521-dd4091e9a8bb","Type":"ContainerDied","Data":"db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8"}
Sep 17 23:13:55.201395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.201013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-c44x2" event={"ID":"dfbfe511-148f-437d-b521-dd4091e9a8bb","Type":"ContainerDied","Data":"213f3fef5e2cea35c0f67090f995f12570d05e8995e3302aad7dfa285784bd5d"}
Sep 17 23:13:55.201395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.201028    2580 scope.go:117] "RemoveContainer" containerID="db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8"
Sep 17 23:13:55.209487 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.209463    2580 scope.go:117] "RemoveContainer" containerID="db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8"
Sep 17 23:13:55.209732 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:13:55.209715    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8\": container with ID starting with db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8 not found: ID does not exist" containerID="db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8"
Sep 17 23:13:55.209822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.209744    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8"} err="failed to get container status \"db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8\": rpc error: code = NotFound desc = could not find container \"db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8\": container with ID starting with db28cac93937424eb10cc8e800098adc332105a7ed86a366de7748865d5f5cc8 not found: ID does not exist"
Sep 17 23:13:55.222235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.222211    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c44x2"]
Sep 17 23:13:55.224659 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:55.224637    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-c44x2"]
Sep 17 23:13:56.107481 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:13:56.107451    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfbfe511-148f-437d-b521-dd4091e9a8bb" path="/var/lib/kubelet/pods/dfbfe511-148f-437d-b521-dd4091e9a8bb/volumes"
Sep 17 23:14:03.287111 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:03.287067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:03.394664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:03.390214    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:03.586775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:03.586686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:03.808222 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:03.808193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:15.465164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.465129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:15.691127 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.691092    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-nw5mb"]
Sep 17 23:14:15.691481 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.691465    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbfe511-148f-437d-b521-dd4091e9a8bb" containerName="authorino"
Sep 17 23:14:15.691533 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.691484    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbfe511-148f-437d-b521-dd4091e9a8bb" containerName="authorino"
Sep 17 23:14:15.691567 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.691553    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbfe511-148f-437d-b521-dd4091e9a8bb" containerName="authorino"
Sep 17 23:14:15.696148 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.696128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw5mb"
Sep 17 23:14:15.698477 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.698456    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cgpp7\""
Sep 17 23:14:15.702842 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.702819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw5mb"]
Sep 17 23:14:15.793235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.793109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjpbz\" (UniqueName: \"kubernetes.io/projected/3af85225-953e-45d7-ac7a-2527d3063fba-kube-api-access-bjpbz\") pod \"authorino-557f876f69-nw5mb\" (UID: \"3af85225-953e-45d7-ac7a-2527d3063fba\") " pod="kuadrant-system/authorino-557f876f69-nw5mb"
Sep 17 23:14:15.869965 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.869933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:15.894335 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.894301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjpbz\" (UniqueName: \"kubernetes.io/projected/3af85225-953e-45d7-ac7a-2527d3063fba-kube-api-access-bjpbz\") pod \"authorino-557f876f69-nw5mb\" (UID: \"3af85225-953e-45d7-ac7a-2527d3063fba\") " pod="kuadrant-system/authorino-557f876f69-nw5mb"
Sep 17 23:14:15.907290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.907264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjpbz\" (UniqueName: \"kubernetes.io/projected/3af85225-953e-45d7-ac7a-2527d3063fba-kube-api-access-bjpbz\") pod \"authorino-557f876f69-nw5mb\" (UID: \"3af85225-953e-45d7-ac7a-2527d3063fba\") " pod="kuadrant-system/authorino-557f876f69-nw5mb"
Sep 17 23:14:15.940265 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.940234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw5mb"]
Sep 17 23:14:15.940476 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.940464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw5mb"
Sep 17 23:14:15.974196 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.974158    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d898bcc85-s7tdn"]
Sep 17 23:14:15.979602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.979576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:15.989543 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:15.989520    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 23:14:16.007815 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.007757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d898bcc85-s7tdn"]
Sep 17 23:14:16.095743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.095642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjt7\" (UniqueName: \"kubernetes.io/projected/ee15df16-ffba-4eef-bc6e-28bcfe840069-kube-api-access-7mjt7\") pod \"authorino-6d898bcc85-s7tdn\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") " pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:16.095743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.095709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee15df16-ffba-4eef-bc6e-28bcfe840069-tls-cert\") pod \"authorino-6d898bcc85-s7tdn\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") " pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:16.197194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.197156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjt7\" (UniqueName: \"kubernetes.io/projected/ee15df16-ffba-4eef-bc6e-28bcfe840069-kube-api-access-7mjt7\") pod \"authorino-6d898bcc85-s7tdn\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") " pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:16.197376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.197356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee15df16-ffba-4eef-bc6e-28bcfe840069-tls-cert\") pod \"authorino-6d898bcc85-s7tdn\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") " pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:16.199967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.199945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee15df16-ffba-4eef-bc6e-28bcfe840069-tls-cert\") pod \"authorino-6d898bcc85-s7tdn\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") " pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:16.205837 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.205784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjt7\" (UniqueName: \"kubernetes.io/projected/ee15df16-ffba-4eef-bc6e-28bcfe840069-kube-api-access-7mjt7\") pod \"authorino-6d898bcc85-s7tdn\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") " pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:16.214901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.214846    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw5mb"]
Sep 17 23:14:16.216941 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:14:16.216914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3af85225_953e_45d7_ac7a_2527d3063fba.slice/crio-53046f985daaaea3ab044a979938bd521972b495cd82f1e10313de237b2faf95 WatchSource:0}: Error finding container 53046f985daaaea3ab044a979938bd521972b495cd82f1e10313de237b2faf95: Status 404 returned error can't find the container with id 53046f985daaaea3ab044a979938bd521972b495cd82f1e10313de237b2faf95
Sep 17 23:14:16.289991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.289965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:14:16.296165 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.296138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nw5mb" event={"ID":"3af85225-953e-45d7-ac7a-2527d3063fba","Type":"ContainerStarted","Data":"53046f985daaaea3ab044a979938bd521972b495cd82f1e10313de237b2faf95"}
Sep 17 23:14:16.557967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:16.557942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d898bcc85-s7tdn"]
Sep 17 23:14:16.560445 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:14:16.560411    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee15df16_ffba_4eef_bc6e_28bcfe840069.slice/crio-34a89718bb04b98ba5d34dd905b1729456007a4f7e5a03bf25038fed676ba529 WatchSource:0}: Error finding container 34a89718bb04b98ba5d34dd905b1729456007a4f7e5a03bf25038fed676ba529: Status 404 returned error can't find the container with id 34a89718bb04b98ba5d34dd905b1729456007a4f7e5a03bf25038fed676ba529
Sep 17 23:14:17.302079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.302040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d898bcc85-s7tdn" event={"ID":"ee15df16-ffba-4eef-bc6e-28bcfe840069","Type":"ContainerStarted","Data":"1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be"}
Sep 17 23:14:17.302265 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.302089    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d898bcc85-s7tdn" event={"ID":"ee15df16-ffba-4eef-bc6e-28bcfe840069","Type":"ContainerStarted","Data":"34a89718bb04b98ba5d34dd905b1729456007a4f7e5a03bf25038fed676ba529"}
Sep 17 23:14:17.303619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.303596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nw5mb" event={"ID":"3af85225-953e-45d7-ac7a-2527d3063fba","Type":"ContainerStarted","Data":"5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972"}
Sep 17 23:14:17.303715 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.303661    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-nw5mb" podUID="3af85225-953e-45d7-ac7a-2527d3063fba" containerName="authorino" containerID="cri-o://5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972" gracePeriod=30
Sep 17 23:14:17.319080 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.319032    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d898bcc85-s7tdn" podStartSLOduration=1.764661087 podStartE2EDuration="2.31901364s" totalImagesPullingTime="554.352553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:17.316792672 +0000 UTC m=+467.746652515" watchObservedRunningTime="2026-09-17 23:14:17.31901364 +0000 UTC m=+467.748873456"
Sep 17 23:14:17.332315 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.332258    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-nw5mb" podStartSLOduration=1.724501931 podStartE2EDuration="2.332239342s" totalImagesPullingTime="607.737411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:17.33157636 +0000 UTC m=+467.761436178" watchObservedRunningTime="2026-09-17 23:14:17.332239342 +0000 UTC m=+467.762099161"
Sep 17 23:14:17.541664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.541641    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw5mb"
Sep 17 23:14:17.711374 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.711284    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjpbz\" (UniqueName: \"kubernetes.io/projected/3af85225-953e-45d7-ac7a-2527d3063fba-kube-api-access-bjpbz\") pod \"3af85225-953e-45d7-ac7a-2527d3063fba\" (UID: \"3af85225-953e-45d7-ac7a-2527d3063fba\") "
Sep 17 23:14:17.713508 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.713484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af85225-953e-45d7-ac7a-2527d3063fba-kube-api-access-bjpbz" (OuterVolumeSpecName: "kube-api-access-bjpbz") pod "3af85225-953e-45d7-ac7a-2527d3063fba" (UID: "3af85225-953e-45d7-ac7a-2527d3063fba"). InnerVolumeSpecName "kube-api-access-bjpbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:17.811929 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:17.811899    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjpbz\" (UniqueName: \"kubernetes.io/projected/3af85225-953e-45d7-ac7a-2527d3063fba-kube-api-access-bjpbz\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:14:18.309654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.309618    2580 generic.go:356] "Generic (PLEG): container finished" podID="3af85225-953e-45d7-ac7a-2527d3063fba" containerID="5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972" exitCode=0
Sep 17 23:14:18.309881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.309661    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw5mb"
Sep 17 23:14:18.309881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.309722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nw5mb" event={"ID":"3af85225-953e-45d7-ac7a-2527d3063fba","Type":"ContainerDied","Data":"5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972"}
Sep 17 23:14:18.309881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.309763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-nw5mb" event={"ID":"3af85225-953e-45d7-ac7a-2527d3063fba","Type":"ContainerDied","Data":"53046f985daaaea3ab044a979938bd521972b495cd82f1e10313de237b2faf95"}
Sep 17 23:14:18.309881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.309782    2580 scope.go:117] "RemoveContainer" containerID="5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972"
Sep 17 23:14:18.318135 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.318118    2580 scope.go:117] "RemoveContainer" containerID="5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972"
Sep 17 23:14:18.318404 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:14:18.318390    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972\": container with ID starting with 5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972 not found: ID does not exist" containerID="5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972"
Sep 17 23:14:18.318462 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.318410    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972"} err="failed to get container status \"5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972\": rpc error: code = NotFound desc = could not find container \"5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972\": container with ID starting with 5c2c325b6d009855d9632e30bc612fa2f9e8d769bcf801492c6969936d760972 not found: ID does not exist"
Sep 17 23:14:18.326177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.326154    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw5mb"]
Sep 17 23:14:18.329346 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:18.329325    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw5mb"]
Sep 17 23:14:20.108570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:20.108535    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af85225-953e-45d7-ac7a-2527d3063fba" path="/var/lib/kubelet/pods/3af85225-953e-45d7-ac7a-2527d3063fba/volumes"
Sep 17 23:14:22.252749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:22.252718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:22.348034 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:22.348004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:22.403533 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:22.403504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:22.469055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:22.469025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:22.621889 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:22.621794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:25.752365 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:25.752331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:25.850425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:25.850393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:43.988965 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:43.988929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:54.153775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:54.153735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:54.253554 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:54.253516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:55.541746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:55.541715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:55.613671 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:55.613639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:55.686288 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:55.686259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:55.786762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:55.786729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:56.013075 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.013042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:56.112908 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.112880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:56.207295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.207261    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"]
Sep 17 23:14:56.207586 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.207574    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af85225-953e-45d7-ac7a-2527d3063fba" containerName="authorino"
Sep 17 23:14:56.207640 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.207587    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af85225-953e-45d7-ac7a-2527d3063fba" containerName="authorino"
Sep 17 23:14:56.207684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.207650    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af85225-953e-45d7-ac7a-2527d3063fba" containerName="authorino"
Sep 17 23:14:56.210504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.210486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.212981 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.212958    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 23:14:56.213079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.212979    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lmktb\""
Sep 17 23:14:56.221169 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.221146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"]
Sep 17 23:14:56.326635 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.326545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdh62\" (UniqueName: \"kubernetes.io/projected/cad23c97-6ac2-4db6-b447-2adaa97042e1-kube-api-access-tdh62\") pod \"payload-pre-processing-59fbc47bf8-x9jm5\" (UID: \"cad23c97-6ac2-4db6-b447-2adaa97042e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.326845 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.326637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad23c97-6ac2-4db6-b447-2adaa97042e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x9jm5\" (UID: \"cad23c97-6ac2-4db6-b447-2adaa97042e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.427352 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.427317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad23c97-6ac2-4db6-b447-2adaa97042e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x9jm5\" (UID: \"cad23c97-6ac2-4db6-b447-2adaa97042e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.427551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.427436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdh62\" (UniqueName: \"kubernetes.io/projected/cad23c97-6ac2-4db6-b447-2adaa97042e1-kube-api-access-tdh62\") pod \"payload-pre-processing-59fbc47bf8-x9jm5\" (UID: \"cad23c97-6ac2-4db6-b447-2adaa97042e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.428339 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.428314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cad23c97-6ac2-4db6-b447-2adaa97042e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x9jm5\" (UID: \"cad23c97-6ac2-4db6-b447-2adaa97042e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.440175 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.440139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdh62\" (UniqueName: \"kubernetes.io/projected/cad23c97-6ac2-4db6-b447-2adaa97042e1-kube-api-access-tdh62\") pod \"payload-pre-processing-59fbc47bf8-x9jm5\" (UID: \"cad23c97-6ac2-4db6-b447-2adaa97042e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.521355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.521310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:56.812608 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:56.812562    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"]
Sep 17 23:14:56.816610 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:14:56.816574    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad23c97_6ac2_4db6_b447_2adaa97042e1.slice/crio-e711c14d0682bd5fc55d483ec79de615438b00f193334b6459308fde95a2ca18 WatchSource:0}: Error finding container e711c14d0682bd5fc55d483ec79de615438b00f193334b6459308fde95a2ca18: Status 404 returned error can't find the container with id e711c14d0682bd5fc55d483ec79de615438b00f193334b6459308fde95a2ca18
Sep 17 23:14:57.100092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:57.100063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:57.200178 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:57.200149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:57.496419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:57.496379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5" event={"ID":"cad23c97-6ac2-4db6-b447-2adaa97042e1","Type":"ContainerStarted","Data":"e711c14d0682bd5fc55d483ec79de615438b00f193334b6459308fde95a2ca18"}
Sep 17 23:14:58.502447 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:58.502397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5" event={"ID":"cad23c97-6ac2-4db6-b447-2adaa97042e1","Type":"ContainerStarted","Data":"636402cbdfebe502661d912e42141ee8a88f22ae7c86506693600500a558b25b"}
Sep 17 23:14:58.502873 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:58.502466    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:14:58.518877 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:14:58.518822    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5" podStartSLOduration=1.372380337 podStartE2EDuration="2.518779626s" totalImagesPullingTime="1.146399289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:58.518328766 +0000 UTC m=+508.948188585" watchObservedRunningTime="2026-09-17 23:14:58.518779626 +0000 UTC m=+508.948639447"
Sep 17 23:15:00.132116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:00.132087    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"]
Sep 17 23:15:00.135590 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:00.135573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"
Sep 17 23:15:00.137792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:00.137770    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:15:00.144732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:00.144708    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"]
Sep 17 23:15:00.146327 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:00.146302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"
Sep 17 23:15:00.422856 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:00.422827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"]
Sep 17 23:15:00.425563 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:15:00.425519    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc802ed27_3127_4489_875f_e574380892f0.slice/crio-6ca6363dbf0496ab332d557cf7ff0dea99c9cd11a4e13084b22378b6ad99de48 WatchSource:0}: Error finding container 6ca6363dbf0496ab332d557cf7ff0dea99c9cd11a4e13084b22378b6ad99de48: Status 404 returned error can't find the container with id 6ca6363dbf0496ab332d557cf7ff0dea99c9cd11a4e13084b22378b6ad99de48
Sep 17 23:15:00.512701 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:00.512667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" event={"ID":"c802ed27-3127-4489-875f-e574380892f0","Type":"ContainerStarted","Data":"6ca6363dbf0496ab332d557cf7ff0dea99c9cd11a4e13084b22378b6ad99de48"}
Sep 17 23:15:03.529792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:03.529755    2580 generic.go:356] "Generic (PLEG): container finished" podID="c802ed27-3127-4489-875f-e574380892f0" containerID="0c6a2fc817420ad253c063e4b41e357befbacc938d6b5ad94efee2b02d8db698" exitCode=7
Sep 17 23:15:03.530320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:03.529845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" event={"ID":"c802ed27-3127-4489-875f-e574380892f0","Type":"ContainerDied","Data":"0c6a2fc817420ad253c063e4b41e357befbacc938d6b5ad94efee2b02d8db698"}
Sep 17 23:15:03.530320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:03.530024    2580 scope.go:117] "RemoveContainer" containerID="0c6a2fc817420ad253c063e4b41e357befbacc938d6b5ad94efee2b02d8db698"
Sep 17 23:15:04.536360 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:04.536273    2580 generic.go:356] "Generic (PLEG): container finished" podID="c802ed27-3127-4489-875f-e574380892f0" containerID="13e79a23a0493f008e72be069bf9fb81686003bfdd0de70813f943ed9ffdb9b1" exitCode=7
Sep 17 23:15:04.536360 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:04.536341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" event={"ID":"c802ed27-3127-4489-875f-e574380892f0","Type":"ContainerDied","Data":"13e79a23a0493f008e72be069bf9fb81686003bfdd0de70813f943ed9ffdb9b1"}
Sep 17 23:15:04.536832 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:04.536403    2580 scope.go:117] "RemoveContainer" containerID="0c6a2fc817420ad253c063e4b41e357befbacc938d6b5ad94efee2b02d8db698"
Sep 17 23:15:04.536832 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:04.536527    2580 scope.go:117] "RemoveContainer" containerID="13e79a23a0493f008e72be069bf9fb81686003bfdd0de70813f943ed9ffdb9b1"
Sep 17 23:15:04.536832 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:15:04.536752    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29828115-tx62h_odh-ai-gateway-infra(c802ed27-3127-4489-875f-e574380892f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" podUID="c802ed27-3127-4489-875f-e574380892f0"
Sep 17 23:15:05.542339 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:05.542315    2580 scope.go:117] "RemoveContainer" containerID="13e79a23a0493f008e72be069bf9fb81686003bfdd0de70813f943ed9ffdb9b1"
Sep 17 23:15:05.542711 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:15:05.542520    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29828115-tx62h_odh-ai-gateway-infra(c802ed27-3127-4489-875f-e574380892f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" podUID="c802ed27-3127-4489-875f-e574380892f0"
Sep 17 23:15:06.365961 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:06.365923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:06.460427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:06.460390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:09.509182 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:09.509147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x9jm5"
Sep 17 23:15:17.102826 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:17.102767    2580 scope.go:117] "RemoveContainer" containerID="13e79a23a0493f008e72be069bf9fb81686003bfdd0de70813f943ed9ffdb9b1"
Sep 17 23:15:17.599511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:17.599475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" event={"ID":"c802ed27-3127-4489-875f-e574380892f0","Type":"ContainerStarted","Data":"8b0f327d3d22a96921a769280a48c53e881471c064fd29fc7943d91d4749c7e2"}
Sep 17 23:15:17.614929 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:17.614869    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" podStartSLOduration=15.246342342 podStartE2EDuration="17.614850192s" totalImagesPullingTime="2.36850785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:17.613287296 +0000 UTC m=+528.043147129" watchObservedRunningTime="2026-09-17 23:15:17.614850192 +0000 UTC m=+528.044710011"
Sep 17 23:15:18.606335 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:18.606302    2580 generic.go:356] "Generic (PLEG): container finished" podID="c802ed27-3127-4489-875f-e574380892f0" containerID="8b0f327d3d22a96921a769280a48c53e881471c064fd29fc7943d91d4749c7e2" exitCode=0
Sep 17 23:15:18.606782 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:18.606356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" event={"ID":"c802ed27-3127-4489-875f-e574380892f0","Type":"ContainerDied","Data":"8b0f327d3d22a96921a769280a48c53e881471c064fd29fc7943d91d4749c7e2"}
Sep 17 23:15:18.606782 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:18.606416    2580 scope.go:117] "RemoveContainer" containerID="13e79a23a0493f008e72be069bf9fb81686003bfdd0de70813f943ed9ffdb9b1"
Sep 17 23:15:18.622636 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:18.622611    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"]
Sep 17 23:15:19.743147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:19.743125    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"
Sep 17 23:15:20.619025 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:20.618974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h" event={"ID":"c802ed27-3127-4489-875f-e574380892f0","Type":"ContainerDied","Data":"6ca6363dbf0496ab332d557cf7ff0dea99c9cd11a4e13084b22378b6ad99de48"}
Sep 17 23:15:20.619025 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:20.619013    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"
Sep 17 23:15:20.619228 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:20.619063    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ca6363dbf0496ab332d557cf7ff0dea99c9cd11a4e13084b22378b6ad99de48"
Sep 17 23:15:20.650150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:20.650119    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"]
Sep 17 23:15:20.657550 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:20.657514    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828115-tx62h"]
Sep 17 23:15:22.108660 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:22.108586    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c802ed27-3127-4489-875f-e574380892f0" path="/var/lib/kubelet/pods/c802ed27-3127-4489-875f-e574380892f0/volumes"
Sep 17 23:15:25.133866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:25.133830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:34.578793 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:34.578758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:37.942607 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.942572    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"]
Sep 17 23:15:37.943116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.943099    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:15:37.943173 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.943119    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:15:37.943173 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.943159    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:15:37.943173 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.943168    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:15:37.943279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.943247    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:15:37.943279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.943266    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:15:37.951382 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.951362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:37.954533 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.954507    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 23:15:37.955109 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.955088    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-frvp9\""
Sep 17 23:15:37.955244 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.955132    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 23:15:37.955244 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.955214    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 17 23:15:37.955564 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.955543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"]
Sep 17 23:15:37.962628 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.962607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:37.984078 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.984039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h594z\" (UniqueName: \"kubernetes.io/projected/994ac596-c94b-420c-8692-29205ba0d9cd-kube-api-access-h594z\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:37.984078 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.984072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:37.984284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.984090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:37.984284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.984185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:37.984284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.984224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:37.984284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:37.984243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ac596-c94b-420c-8692-29205ba0d9cd-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ac596-c94b-420c-8692-29205ba0d9cd-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h594z\" (UniqueName: \"kubernetes.io/projected/994ac596-c94b-420c-8692-29205ba0d9cd-kube-api-access-h594z\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.085980 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.085949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.086162 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.086143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.088576 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.088549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994ac596-c94b-420c-8692-29205ba0d9cd-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.088831 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.088816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994ac596-c94b-420c-8692-29205ba0d9cd-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.093232 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.093210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h594z\" (UniqueName: \"kubernetes.io/projected/994ac596-c94b-420c-8692-29205ba0d9cd-kube-api-access-h594z\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj\" (UID: \"994ac596-c94b-420c-8692-29205ba0d9cd\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.263044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.262945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:38.534839 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.534794    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"]
Sep 17 23:15:38.537645 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:15:38.537606    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994ac596_c94b_420c_8692_29205ba0d9cd.slice/crio-eb385a375360b2878e9dab861b4b2e9d3eb6d13a859dbd3ebb927c9ca6910b1a WatchSource:0}: Error finding container eb385a375360b2878e9dab861b4b2e9d3eb6d13a859dbd3ebb927c9ca6910b1a: Status 404 returned error can't find the container with id eb385a375360b2878e9dab861b4b2e9d3eb6d13a859dbd3ebb927c9ca6910b1a
Sep 17 23:15:38.712507 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:38.712465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj" event={"ID":"994ac596-c94b-420c-8692-29205ba0d9cd","Type":"ContainerStarted","Data":"eb385a375360b2878e9dab861b4b2e9d3eb6d13a859dbd3ebb927c9ca6910b1a"}
Sep 17 23:15:39.720028 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:39.719986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj" event={"ID":"994ac596-c94b-420c-8692-29205ba0d9cd","Type":"ContainerStarted","Data":"5bacab0b1b23e6a22ef36f7c774e48416002040b3edbd2dac98e228e15abf442"}
Sep 17 23:15:39.743401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:39.743345    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj" podStartSLOduration=1.659085031 podStartE2EDuration="2.743326909s" totalImagesPullingTime="1.084241878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:39.742227525 +0000 UTC m=+550.172087344" watchObservedRunningTime="2026-09-17 23:15:39.743326909 +0000 UTC m=+550.173186726"
Sep 17 23:15:48.263947 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:48.263908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:48.264452 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:48.263973    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:48.276527 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:48.276500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:48.770054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:48.770023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj"
Sep 17 23:15:49.549611 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:15:49.549581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:03.966425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:03.966393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:07.438570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:07.438537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:15.340755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.340722    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"]
Sep 17 23:16:15.341164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.341105    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:16:15.341164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.341117    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:16:15.341253 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.341216    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c802ed27-3127-4489-875f-e574380892f0" containerName="cleanup"
Sep 17 23:16:15.351174 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.351149    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.353688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.353663    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 23:16:15.356603 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.356579    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"]
Sep 17 23:16:15.376600 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.376571    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:15.427726 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.427700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.427915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.427740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ea20-7cc8-4f38-b7b0-85502deee9ab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.427915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.427762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.427915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.427833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92lfg\" (UniqueName: \"kubernetes.io/projected/e640ea20-7cc8-4f38-b7b0-85502deee9ab-kube-api-access-92lfg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.427915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.427900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.428062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.427918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529074 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ea20-7cc8-4f38-b7b0-85502deee9ab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92lfg\" (UniqueName: \"kubernetes.io/projected/e640ea20-7cc8-4f38-b7b0-85502deee9ab-kube-api-access-92lfg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.529732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.529637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.531692 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.531674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ea20-7cc8-4f38-b7b0-85502deee9ab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.531971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.531954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ea20-7cc8-4f38-b7b0-85502deee9ab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.537083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.537061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92lfg\" (UniqueName: \"kubernetes.io/projected/e640ea20-7cc8-4f38-b7b0-85502deee9ab-kube-api-access-92lfg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2slcf\" (UID: \"e640ea20-7cc8-4f38-b7b0-85502deee9ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.663050 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.663014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:15.936377 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:15.936352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"]
Sep 17 23:16:15.938635 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:16:15.938606    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode640ea20_7cc8_4f38_b7b0_85502deee9ab.slice/crio-d1bdbad32407dbfb106119bff75182b4671b426bcfe64043027ecac4df0429ef WatchSource:0}: Error finding container d1bdbad32407dbfb106119bff75182b4671b426bcfe64043027ecac4df0429ef: Status 404 returned error can't find the container with id d1bdbad32407dbfb106119bff75182b4671b426bcfe64043027ecac4df0429ef
Sep 17 23:16:16.891883 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:16.891844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf" event={"ID":"e640ea20-7cc8-4f38-b7b0-85502deee9ab","Type":"ContainerStarted","Data":"119206ade70e0cf9b4bcefb837ec9671bd216355ab136294c3e34c181f816ff6"}
Sep 17 23:16:16.892311 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:16.891929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf" event={"ID":"e640ea20-7cc8-4f38-b7b0-85502deee9ab","Type":"ContainerStarted","Data":"d1bdbad32407dbfb106119bff75182b4671b426bcfe64043027ecac4df0429ef"}
Sep 17 23:16:16.915224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:16.915163    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf" podStartSLOduration=1.745769854 podStartE2EDuration="1.915148004s" totalImagesPullingTime="169.37815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:16.914017586 +0000 UTC m=+587.343877405" watchObservedRunningTime="2026-09-17 23:16:16.915148004 +0000 UTC m=+587.345007823"
Sep 17 23:16:21.529486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.529454    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"]
Sep 17 23:16:21.533277 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.533251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.535753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.535733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 23:16:21.544448 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.544427    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"]
Sep 17 23:16:21.559732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.559679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:21.584478 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.584451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11f00a4-8878-4361-9314-f20e96f4b0ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.584619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.584491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.584619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.584538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4m2\" (UniqueName: \"kubernetes.io/projected/e11f00a4-8878-4361-9314-f20e96f4b0ff-kube-api-access-wd4m2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.584619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.584590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.584781 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.584646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.584781 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.584714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.685792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.685754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.685827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11f00a4-8878-4361-9314-f20e96f4b0ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.685865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.685897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4m2\" (UniqueName: \"kubernetes.io/projected/e11f00a4-8878-4361-9314-f20e96f4b0ff-kube-api-access-wd4m2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.685942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.685968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.686243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.686332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.686413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.686391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.688275 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.688252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11f00a4-8878-4361-9314-f20e96f4b0ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.688965 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.688944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11f00a4-8878-4361-9314-f20e96f4b0ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.694018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.693996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4m2\" (UniqueName: \"kubernetes.io/projected/e11f00a4-8878-4361-9314-f20e96f4b0ff-kube-api-access-wd4m2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx\" (UID: \"e11f00a4-8878-4361-9314-f20e96f4b0ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:21.842694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:21.842598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:22.124260 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:22.124233    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"]
Sep 17 23:16:22.126075 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:16:22.126046    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode11f00a4_8878_4361_9314_f20e96f4b0ff.slice/crio-9ba369821e4feabed67f1ae1e45d40ed7854edb40d97ed274d5b630a96567029 WatchSource:0}: Error finding container 9ba369821e4feabed67f1ae1e45d40ed7854edb40d97ed274d5b630a96567029: Status 404 returned error can't find the container with id 9ba369821e4feabed67f1ae1e45d40ed7854edb40d97ed274d5b630a96567029
Sep 17 23:16:22.924962 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:22.924926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx" event={"ID":"e11f00a4-8878-4361-9314-f20e96f4b0ff","Type":"ContainerStarted","Data":"30c33bb2a711a8e89a2b77f95483117eb0937536049ead907272a5a0cfa8a05d"}
Sep 17 23:16:22.924962 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:22.924967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx" event={"ID":"e11f00a4-8878-4361-9314-f20e96f4b0ff","Type":"ContainerStarted","Data":"9ba369821e4feabed67f1ae1e45d40ed7854edb40d97ed274d5b630a96567029"}
Sep 17 23:16:22.943031 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:22.942972    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx" podStartSLOduration=1.780954951 podStartE2EDuration="1.942955679s" totalImagesPullingTime="162.000728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:22.942820859 +0000 UTC m=+593.372680668" watchObservedRunningTime="2026-09-17 23:16:22.942955679 +0000 UTC m=+593.372815496"
Sep 17 23:16:25.663264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:25.663232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:25.663660 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:25.663293    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:25.676097 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:25.676071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:25.950852 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:25.950746    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2slcf"
Sep 17 23:16:30.026179 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:30.026150    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:16:30.029629 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:30.029607    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:16:31.843656 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:31.843602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:31.844157 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:31.843681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:31.856127 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:31.856102    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:16:31.977482 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:16:31.977446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx"
Sep 17 23:17:03.174005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:03.173896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:22.301045 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.300998    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d898bcc85-s7tdn"]
Sep 17 23:17:22.301512 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.301285    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d898bcc85-s7tdn" podUID="ee15df16-ffba-4eef-bc6e-28bcfe840069" containerName="authorino" containerID="cri-o://1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be" gracePeriod=30
Sep 17 23:17:22.548698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.548669    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:17:22.654818 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.654780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mjt7\" (UniqueName: \"kubernetes.io/projected/ee15df16-ffba-4eef-bc6e-28bcfe840069-kube-api-access-7mjt7\") pod \"ee15df16-ffba-4eef-bc6e-28bcfe840069\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") "
Sep 17 23:17:22.654993 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.654831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee15df16-ffba-4eef-bc6e-28bcfe840069-tls-cert\") pod \"ee15df16-ffba-4eef-bc6e-28bcfe840069\" (UID: \"ee15df16-ffba-4eef-bc6e-28bcfe840069\") "
Sep 17 23:17:22.657040 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.657013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee15df16-ffba-4eef-bc6e-28bcfe840069-kube-api-access-7mjt7" (OuterVolumeSpecName: "kube-api-access-7mjt7") pod "ee15df16-ffba-4eef-bc6e-28bcfe840069" (UID: "ee15df16-ffba-4eef-bc6e-28bcfe840069"). InnerVolumeSpecName "kube-api-access-7mjt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:17:22.664728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.664705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee15df16-ffba-4eef-bc6e-28bcfe840069-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ee15df16-ffba-4eef-bc6e-28bcfe840069" (UID: "ee15df16-ffba-4eef-bc6e-28bcfe840069"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:17:22.755902 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.755871    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjt7\" (UniqueName: \"kubernetes.io/projected/ee15df16-ffba-4eef-bc6e-28bcfe840069-kube-api-access-7mjt7\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:17:22.756072 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.755919    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee15df16-ffba-4eef-bc6e-28bcfe840069-tls-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:17:22.798515 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.798483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:22.893955 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:22.893921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:23.216743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.216714    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee15df16-ffba-4eef-bc6e-28bcfe840069" containerID="1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be" exitCode=0
Sep 17 23:17:23.216939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.216759    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d898bcc85-s7tdn"
Sep 17 23:17:23.216939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.216770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d898bcc85-s7tdn" event={"ID":"ee15df16-ffba-4eef-bc6e-28bcfe840069","Type":"ContainerDied","Data":"1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be"}
Sep 17 23:17:23.216939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.216829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d898bcc85-s7tdn" event={"ID":"ee15df16-ffba-4eef-bc6e-28bcfe840069","Type":"ContainerDied","Data":"34a89718bb04b98ba5d34dd905b1729456007a4f7e5a03bf25038fed676ba529"}
Sep 17 23:17:23.216939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.216850    2580 scope.go:117] "RemoveContainer" containerID="1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be"
Sep 17 23:17:23.225355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.225338    2580 scope.go:117] "RemoveContainer" containerID="1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be"
Sep 17 23:17:23.225654 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:17:23.225634    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be\": container with ID starting with 1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be not found: ID does not exist" containerID="1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be"
Sep 17 23:17:23.225702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.225663    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be"} err="failed to get container status \"1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be\": rpc error: code = NotFound desc = could not find container \"1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be\": container with ID starting with 1e551dbe6f3c08d8c19c39a5e5db6eed11b8006d8308aee11647a973b528a4be not found: ID does not exist"
Sep 17 23:17:23.237316 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.237284    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d898bcc85-s7tdn"]
Sep 17 23:17:23.242947 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.242925    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d898bcc85-s7tdn"]
Sep 17 23:17:23.909391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:23.909347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:24.003237 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:24.003207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:24.110189 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:24.110148    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee15df16-ffba-4eef-bc6e-28bcfe840069" path="/var/lib/kubelet/pods/ee15df16-ffba-4eef-bc6e-28bcfe840069/volumes"
Sep 17 23:17:39.287197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.287161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:39.387465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.387435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:39.487120 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.487086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:39.652246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.652211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:39.751522 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.751493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:39.771851 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.771788    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"]
Sep 17 23:17:39.772180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.772168    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee15df16-ffba-4eef-bc6e-28bcfe840069" containerName="authorino"
Sep 17 23:17:39.772224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.772182    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee15df16-ffba-4eef-bc6e-28bcfe840069" containerName="authorino"
Sep 17 23:17:39.772261 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.772238    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee15df16-ffba-4eef-bc6e-28bcfe840069" containerName="authorino"
Sep 17 23:17:39.776630 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.776610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.779145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.779118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0b4d93-openshift-default-dockercfg-vh7vl\""
Sep 17 23:17:39.779275 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.779128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 23:17:39.790252 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.790225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"]
Sep 17 23:17:39.855432 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.855400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:39.917766 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.917672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-podinfo\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.917766 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.917712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-token\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.918015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.917826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-certs\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.918015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.917881    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-data\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.918015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.917931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-envoy\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.918015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.917964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-socket\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.918015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.917982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istiod-ca-cert\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.918244 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.918041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-credential-socket\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.918244 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.918080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nxs\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-kube-api-access-b6nxs\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:39.995976 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:39.995939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:40.019165 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-socket\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019165 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istiod-ca-cert\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-credential-socket\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nxs\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-kube-api-access-b6nxs\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019547 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-podinfo\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019604 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-token\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019604 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-certs\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-socket\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-data\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-envoy\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-credential-socket\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.019972 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.019937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-certs\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.020041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.020024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-data\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.020102 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.020082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istiod-ca-cert\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.022095 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.022069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-envoy\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.022400 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.022377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-podinfo\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.027246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.027223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-token\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.027606 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.027587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nxs\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-kube-api-access-b6nxs\") pod \"e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") " pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.089815 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.089765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:40.093529 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.093499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:40.366026 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.365936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"]
Sep 17 23:17:40.368338 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:17:40.368312    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c76884_ca63_40e3_b4d0_9a7f9a71d691.slice/crio-47e5e286a8f9f44b0cd64635e827abde657710de1fe4dfa4869ab1d4b587945a WatchSource:0}: Error finding container 47e5e286a8f9f44b0cd64635e827abde657710de1fe4dfa4869ab1d4b587945a: Status 404 returned error can't find the container with id 47e5e286a8f9f44b0cd64635e827abde657710de1fe4dfa4869ab1d4b587945a
Sep 17 23:17:40.705017 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.704983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:40.804760 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:40.804727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:41.157695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:41.157657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:41.306925 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:41.306888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" event={"ID":"27c76884-ca63-40e3-b4d0-9a7f9a71d691","Type":"ContainerStarted","Data":"47e5e286a8f9f44b0cd64635e827abde657710de1fe4dfa4869ab1d4b587945a"}
Sep 17 23:17:42.316494 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:42.316457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:42.425970 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:42.425936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:42.534267 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:42.534229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:42.627270 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:42.627179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:43.076016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.075978    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:17:43.076098 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.076057    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:17:43.076098 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.076086    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:17:43.319790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.319754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" event={"ID":"27c76884-ca63-40e3-b4d0-9a7f9a71d691","Type":"ContainerStarted","Data":"ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71"}
Sep 17 23:17:43.347137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.347035    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podStartSLOduration=1.6417311190000001 podStartE2EDuration="4.347011351s" totalImagesPullingTime="2.705280232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:43.345771009 +0000 UTC m=+673.775630840" watchObservedRunningTime="2026-09-17 23:17:43.347011351 +0000 UTC m=+673.776871168"
Sep 17 23:17:43.676901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.676867    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:43.773695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.773664    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:43.873774 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:43.873737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:44.090856 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.090748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:44.095891 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.095868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:44.193361 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.193326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:44.268315 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.268278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:44.308093 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.308060    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"]
Sep 17 23:17:44.311736 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.311719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.314066 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.314045    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7ab633\""
Sep 17 23:17:44.322331 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.322308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"]
Sep 17 23:17:44.325387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.325356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:44.326471 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.326451    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:17:44.335334 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.335312    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"]
Sep 17 23:17:44.338819 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.338780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.341563 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.341511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7ab633-dockercfg-tzvph\""
Sep 17 23:17:44.351139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.351119    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"]
Sep 17 23:17:44.461337 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.461307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzsfq\" (UniqueName: \"kubernetes.io/projected/a8d687b9-8e77-442e-8522-0532fd561fa0-kube-api-access-gzsfq\") pod \"payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.461511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.461370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d687b9-8e77-442e-8522-0532fd561fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.461511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.461477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqg29\" (UniqueName: \"kubernetes.io/projected/3973213c-acc3-4030-a510-19690f545bd6-kube-api-access-pqg29\") pod \"payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.461511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.461500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973213c-acc3-4030-a510-19690f545bd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.562653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.562620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqg29\" (UniqueName: \"kubernetes.io/projected/3973213c-acc3-4030-a510-19690f545bd6-kube-api-access-pqg29\") pod \"payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.562828 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.562664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973213c-acc3-4030-a510-19690f545bd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.562828 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.562751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzsfq\" (UniqueName: \"kubernetes.io/projected/a8d687b9-8e77-442e-8522-0532fd561fa0-kube-api-access-gzsfq\") pod \"payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.562828 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.562817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d687b9-8e77-442e-8522-0532fd561fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.563492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.563472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973213c-acc3-4030-a510-19690f545bd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.563535 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.563519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d687b9-8e77-442e-8522-0532fd561fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.570909 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.570887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzsfq\" (UniqueName: \"kubernetes.io/projected/a8d687b9-8e77-442e-8522-0532fd561fa0-kube-api-access-gzsfq\") pod \"payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.571046 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.570946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqg29\" (UniqueName: \"kubernetes.io/projected/3973213c-acc3-4030-a510-19690f545bd6-kube-api-access-pqg29\") pod \"payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.623004 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.622918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:44.649137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.649102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:44.715360 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.715304    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:44.815984 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.815955    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:44.902836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.902785    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"]
Sep 17 23:17:44.904476 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:17:44.904454    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d687b9_8e77_442e_8522_0532fd561fa0.slice/crio-f041c879fbd9dc1d62b9693487fe7f804f4273824e3b202c9b884101dfbf1a61 WatchSource:0}: Error finding container f041c879fbd9dc1d62b9693487fe7f804f4273824e3b202c9b884101dfbf1a61: Status 404 returned error can't find the container with id f041c879fbd9dc1d62b9693487fe7f804f4273824e3b202c9b884101dfbf1a61
Sep 17 23:17:44.944504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.944481    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"]
Sep 17 23:17:44.946433 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:17:44.946406    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3973213c_acc3_4030_a510_19690f545bd6.slice/crio-fb77c71900391127568217ad2c14b86f5690a479c532098847fbed971d5f5178 WatchSource:0}: Error finding container fb77c71900391127568217ad2c14b86f5690a479c532098847fbed971d5f5178: Status 404 returned error can't find the container with id fb77c71900391127568217ad2c14b86f5690a479c532098847fbed971d5f5178
Sep 17 23:17:44.965390 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:44.965360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.325153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.325066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.331047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.331014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98" event={"ID":"3973213c-acc3-4030-a510-19690f545bd6","Type":"ContainerStarted","Data":"29bcadf7d2cbf70c41304de5a45d4344b1665fd7b9412bc926bfb4c9fdd2df2c"}
Sep 17 23:17:45.331191 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.331077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98" event={"ID":"3973213c-acc3-4030-a510-19690f545bd6","Type":"ContainerStarted","Data":"fb77c71900391127568217ad2c14b86f5690a479c532098847fbed971d5f5178"}
Sep 17 23:17:45.331249 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.331201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:45.332666 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.332635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb" event={"ID":"a8d687b9-8e77-442e-8522-0532fd561fa0","Type":"ContainerStarted","Data":"fb1f60ef543b4efab67880ec7c8c3e040c7f1a2ff75b05b06ebdeaa6a85da713"}
Sep 17 23:17:45.332817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.332684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb" event={"ID":"a8d687b9-8e77-442e-8522-0532fd561fa0","Type":"ContainerStarted","Data":"f041c879fbd9dc1d62b9693487fe7f804f4273824e3b202c9b884101dfbf1a61"}
Sep 17 23:17:45.332817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.332732    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:45.359364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.359320    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98" podStartSLOduration=1.359304953 podStartE2EDuration="1.359304953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:45.35702796 +0000 UTC m=+675.786887780" watchObservedRunningTime="2026-09-17 23:17:45.359304953 +0000 UTC m=+675.789164771"
Sep 17 23:17:45.375992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.375942    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb" podStartSLOduration=1.3759277619999999 podStartE2EDuration="1.375927762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:45.374522241 +0000 UTC m=+675.804382059" watchObservedRunningTime="2026-09-17 23:17:45.375927762 +0000 UTC m=+675.805787580"
Sep 17 23:17:45.424904 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.424871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.516409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.516363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.682674 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.682626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.741299 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.741270    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"]
Sep 17 23:17:45.744954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.744932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:45.747658 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.747632    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7aa654-dockercfg-dlkjd\""
Sep 17 23:17:45.748166 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.748135    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7aa654\""
Sep 17 23:17:45.755860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.755834    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"]
Sep 17 23:17:45.876888 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.876852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5brf\" (UniqueName: \"kubernetes.io/projected/a1a93a76-0c68-4a48-a983-ea597a78ec2c-kube-api-access-w5brf\") pod \"payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:45.877093 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.877018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a93a76-0c68-4a48-a983-ea597a78ec2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:45.978493 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.978387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a93a76-0c68-4a48-a983-ea597a78ec2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:45.978493 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.978451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5brf\" (UniqueName: \"kubernetes.io/projected/a1a93a76-0c68-4a48-a983-ea597a78ec2c-kube-api-access-w5brf\") pod \"payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:45.979227 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.979201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a93a76-0c68-4a48-a983-ea597a78ec2c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:45.987517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:45.987493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5brf\" (UniqueName: \"kubernetes.io/projected/a1a93a76-0c68-4a48-a983-ea597a78ec2c-kube-api-access-w5brf\") pod \"payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:46.045333 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:46.045297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:46.057270 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:46.057242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:46.340060 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:46.340032    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"]
Sep 17 23:17:46.341078 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:17:46.341052    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a93a76_0c68_4a48_a983_ea597a78ec2c.slice/crio-95869a83b0d81d7b05412c597cf71378a5c3a89b863b344ddff51a2feec1deb4 WatchSource:0}: Error finding container 95869a83b0d81d7b05412c597cf71378a5c3a89b863b344ddff51a2feec1deb4: Status 404 returned error can't find the container with id 95869a83b0d81d7b05412c597cf71378a5c3a89b863b344ddff51a2feec1deb4
Sep 17 23:17:47.039378 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.039343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:47.243205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.243170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:47.344956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.344818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5" event={"ID":"a1a93a76-0c68-4a48-a983-ea597a78ec2c","Type":"ContainerStarted","Data":"8051a969136ca38006a14c5535a010f4f650970d4ad504d566bb3bc5073f1564"}
Sep 17 23:17:47.344956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.344878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5" event={"ID":"a1a93a76-0c68-4a48-a983-ea597a78ec2c","Type":"ContainerStarted","Data":"95869a83b0d81d7b05412c597cf71378a5c3a89b863b344ddff51a2feec1deb4"}
Sep 17 23:17:47.344956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.344918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:47.364625 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.364566    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5" podStartSLOduration=2.364543739 podStartE2EDuration="2.364543739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:47.363433203 +0000 UTC m=+677.793293022" watchObservedRunningTime="2026-09-17 23:17:47.364543739 +0000 UTC m=+677.794403559"
Sep 17 23:17:47.555039 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.554991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:47.851582 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:47.851552    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:48.016150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:48.016094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:48.119174 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:48.119087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:48.302380 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:48.302339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:48.627475 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:48.627444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:48.845298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:48.845268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:49.128217 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:49.128185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:49.194053 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:49.194012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:49.527847 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:49.527795    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:49.966421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:49.966389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:50.066129 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.066097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:50.111179 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.111150    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"]
Sep 17 23:17:50.114640 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.114616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.117151 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.117110    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0b4d93\""
Sep 17 23:17:50.117275 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.117162    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0b4d93-dockercfg-dz6kw\""
Sep 17 23:17:50.128464 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.128441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"]
Sep 17 23:17:50.219829 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.219701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05076899-68ae-4e95-8a3c-a347efb2d140-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.220017 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.219835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq87d\" (UniqueName: \"kubernetes.io/projected/05076899-68ae-4e95-8a3c-a347efb2d140-kube-api-access-dq87d\") pod \"payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.320795 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.320746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05076899-68ae-4e95-8a3c-a347efb2d140-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.320984 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.320892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq87d\" (UniqueName: \"kubernetes.io/projected/05076899-68ae-4e95-8a3c-a347efb2d140-kube-api-access-dq87d\") pod \"payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.321585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.321566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05076899-68ae-4e95-8a3c-a347efb2d140-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.328503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.328466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq87d\" (UniqueName: \"kubernetes.io/projected/05076899-68ae-4e95-8a3c-a347efb2d140-kube-api-access-dq87d\") pod \"payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.367944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.367907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:50.425917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.425881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:50.496659 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.496474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:50.592532 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.592496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:50.701783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:50.701656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"]
Sep 17 23:17:50.704978 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:17:50.704946    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05076899_68ae_4e95_8a3c_a347efb2d140.slice/crio-cd6f37a0cb1b19e225a0472920458f53e1f8a2403a443441bec3a31656bdb408 WatchSource:0}: Error finding container cd6f37a0cb1b19e225a0472920458f53e1f8a2403a443441bec3a31656bdb408: Status 404 returned error can't find the container with id cd6f37a0cb1b19e225a0472920458f53e1f8a2403a443441bec3a31656bdb408
Sep 17 23:17:51.358967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:51.358936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:51.369518 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:51.369489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf" event={"ID":"05076899-68ae-4e95-8a3c-a347efb2d140","Type":"ContainerStarted","Data":"2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7"}
Sep 17 23:17:51.369653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:51.369547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf" event={"ID":"05076899-68ae-4e95-8a3c-a347efb2d140","Type":"ContainerStarted","Data":"cd6f37a0cb1b19e225a0472920458f53e1f8a2403a443441bec3a31656bdb408"}
Sep 17 23:17:51.369653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:51.369571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:17:52.314472 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:52.314433    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:53.363192 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:53.363156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:53.628549 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:53.628465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:53.731751 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:53.731720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:54.563137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:54.563108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:54.666039 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:54.666005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:54.895507 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:54.895477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:55.158681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.158587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:55.249231 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.249176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf" podStartSLOduration=5.249156413 podStartE2EDuration="5.249156413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:51.387059586 +0000 UTC m=+681.816919411" watchObservedRunningTime="2026-09-17 23:17:55.249156413 +0000 UTC m=+685.679016231"
Sep 17 23:17:55.250357 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.250337    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"]
Sep 17 23:17:55.253772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.253733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:55.253899 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.253836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.255970 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.255952    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6114ed\""
Sep 17 23:17:55.265291 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.265270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"]
Sep 17 23:17:55.372665 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.372629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4437ed2-2c09-4b1f-b712-05c2f824f24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.372665 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.372663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthvl\" (UniqueName: \"kubernetes.io/projected/c4437ed2-2c09-4b1f-b712-05c2f824f24f-kube-api-access-bthvl\") pod \"payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.473861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.473729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4437ed2-2c09-4b1f-b712-05c2f824f24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.473861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.473781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthvl\" (UniqueName: \"kubernetes.io/projected/c4437ed2-2c09-4b1f-b712-05c2f824f24f-kube-api-access-bthvl\") pod \"payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.474451 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.474428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4437ed2-2c09-4b1f-b712-05c2f824f24f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.481817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.481771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthvl\" (UniqueName: \"kubernetes.io/projected/c4437ed2-2c09-4b1f-b712-05c2f824f24f-kube-api-access-bthvl\") pod \"payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.559843 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.559784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:55.564765 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.564745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:55.627490 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.627448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:55.866141 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.866114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"]
Sep 17 23:17:55.868087 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:17:55.868055    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4437ed2_2c09_4b1f_b712_05c2f824f24f.slice/crio-543badb03090055f4ae21c6b8c9ec09a28e0727feea090607b5b5c23bb9590e3 WatchSource:0}: Error finding container 543badb03090055f4ae21c6b8c9ec09a28e0727feea090607b5b5c23bb9590e3: Status 404 returned error can't find the container with id 543badb03090055f4ae21c6b8c9ec09a28e0727feea090607b5b5c23bb9590e3
Sep 17 23:17:55.885584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:55.885475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:56.060275 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.060245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:56.169587 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.169549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:56.178890 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.178860    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"]
Sep 17 23:17:56.182667 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.182644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.185541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.185518    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f10a7edb\""
Sep 17 23:17:56.185962 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.185942    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f10a7edb-dockercfg-hmmv7\""
Sep 17 23:17:56.194781 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.194752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"]
Sep 17 23:17:56.282105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.281834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914bdc04-21ad-43fa-80c3-87a4d179c4fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.282105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.281949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnfc\" (UniqueName: \"kubernetes.io/projected/914bdc04-21ad-43fa-80c3-87a4d179c4fc-kube-api-access-xvnfc\") pod \"payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.339666 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.339625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:17:56.339848 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.339714    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:17:56.384716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.383336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnfc\" (UniqueName: \"kubernetes.io/projected/914bdc04-21ad-43fa-80c3-87a4d179c4fc-kube-api-access-xvnfc\") pod \"payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.384716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.383458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914bdc04-21ad-43fa-80c3-87a4d179c4fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.384716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.384412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914bdc04-21ad-43fa-80c3-87a4d179c4fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.400570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.400495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnfc\" (UniqueName: \"kubernetes.io/projected/914bdc04-21ad-43fa-80c3-87a4d179c4fc-kube-api-access-xvnfc\") pod \"payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.417339 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.417300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8" event={"ID":"c4437ed2-2c09-4b1f-b712-05c2f824f24f","Type":"ContainerStarted","Data":"1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e"}
Sep 17 23:17:56.417614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.417590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8" event={"ID":"c4437ed2-2c09-4b1f-b712-05c2f824f24f","Type":"ContainerStarted","Data":"543badb03090055f4ae21c6b8c9ec09a28e0727feea090607b5b5c23bb9590e3"}
Sep 17 23:17:56.418525 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.418502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:17:56.517812 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.517713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:56.680696 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.680660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:56.798185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.795518    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8" podStartSLOduration=1.795492779 podStartE2EDuration="1.795492779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:56.434415259 +0000 UTC m=+686.864275081" watchObservedRunningTime="2026-09-17 23:17:56.795492779 +0000 UTC m=+687.225352598"
Sep 17 23:17:56.798185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:56.795685    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"]
Sep 17 23:17:57.019888 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:57.019856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:57.424860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:57.424823    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j" event={"ID":"914bdc04-21ad-43fa-80c3-87a4d179c4fc","Type":"ContainerStarted","Data":"2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935"}
Sep 17 23:17:57.425042 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:57.424883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j" event={"ID":"914bdc04-21ad-43fa-80c3-87a4d179c4fc","Type":"ContainerStarted","Data":"891bf183dff88be665d8e414af0ebc94236f6bff310df4107a66a99cc0ebb05c"}
Sep 17 23:17:57.425042 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:57.424927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:17:57.478353 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:57.478300    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j" podStartSLOduration=1.478283825 podStartE2EDuration="1.478283825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:57.476159941 +0000 UTC m=+687.906019761" watchObservedRunningTime="2026-09-17 23:17:57.478283825 +0000 UTC m=+687.908143642"
Sep 17 23:17:57.515760 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:57.515716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:57.613911 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:57.613870    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:58.355376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:58.355348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:17:58.860134 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:58.860097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:59.690305 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:17:59.690272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:00.809321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:00.809260    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:01.730612 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:01.730573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:02.130993 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.130907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:02.342729 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.342694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:02.378870 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.378837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:18:02.506739 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.506708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:02.596981 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.596927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:02.758956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.758875    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:02.828460 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.828426    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:02.896184 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:02.896141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:03.854093 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:03.854065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:04.465276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:04.465181    2580 generic.go:356] "Generic (PLEG): container finished" podID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerID="ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71" exitCode=0
Sep 17 23:18:04.465465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:04.465269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" event={"ID":"27c76884-ca63-40e3-b4d0-9a7f9a71d691","Type":"ContainerDied","Data":"ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71"}
Sep 17 23:18:04.465723 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:04.465707    2580 scope.go:117] "RemoveContainer" containerID="ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71"
Sep 17 23:18:04.466987 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:04.466956    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:18:04.467111 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:04.467031    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:18:04.467111 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:04.467066    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:18:04.743535 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:04.743443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:05.065356 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:05.065243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:05.348051 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:05.347965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:05.472179 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:05.472149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" event={"ID":"27c76884-ca63-40e3-b4d0-9a7f9a71d691","Type":"ContainerStarted","Data":"f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae"}
Sep 17 23:18:05.610596 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:05.610496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:05.743956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:05.743921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:05.854985 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:05.854958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:06.091146 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:06.091112    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:18:06.091974 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:06.091152    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 17 23:18:06.092201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:06.092167    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 17 23:18:06.332298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:06.332209    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:07.066015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:07.065964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:07.090683 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:07.090646    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 17 23:18:07.091002 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:07.090970    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 17 23:18:07.471472 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:07.471407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:08.091055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:08.091005    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 17 23:18:08.091351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:08.091319    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 17 23:18:08.434453 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:08.434428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:18:08.434617 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:08.434498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:18:08.448203 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:08.448178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:08.544757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:08.544730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:08.645065 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:08.645027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:09.053367 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.053339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:09.093230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.093195    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 17 23:18:09.093542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.093507    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 17 23:18:09.152053 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.152016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:09.179451 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.179412    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"]
Sep 17 23:18:09.193232 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.193148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.194085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.194041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"]
Sep 17 23:18:09.201151 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.201123    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8fc0d3\""
Sep 17 23:18:09.201357 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.201330    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8fc0d3-dockercfg-z76ch\""
Sep 17 23:18:09.229268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.229233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eeba0b-33ed-4ae2-a820-dac3b240fff7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.229426 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.229280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2t8\" (UniqueName: \"kubernetes.io/projected/65eeba0b-33ed-4ae2-a820-dac3b240fff7-kube-api-access-bm2t8\") pod \"payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.246208 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.246183    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"]
Sep 17 23:18:09.250640 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.250623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.258215 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.258188    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"]
Sep 17 23:18:09.329948 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.329914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2t8\" (UniqueName: \"kubernetes.io/projected/65eeba0b-33ed-4ae2-a820-dac3b240fff7-kube-api-access-bm2t8\") pod \"payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.330153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.329984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rvg\" (UniqueName: \"kubernetes.io/projected/001690c8-6863-414e-9c30-d41dbcfcf11b-kube-api-access-49rvg\") pod \"payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.330153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.330046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001690c8-6863-414e-9c30-d41dbcfcf11b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.330153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.330098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eeba0b-33ed-4ae2-a820-dac3b240fff7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.330795 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.330769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eeba0b-33ed-4ae2-a820-dac3b240fff7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.341505 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.341478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2t8\" (UniqueName: \"kubernetes.io/projected/65eeba0b-33ed-4ae2-a820-dac3b240fff7-kube-api-access-bm2t8\") pod \"payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.431331 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.431293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rvg\" (UniqueName: \"kubernetes.io/projected/001690c8-6863-414e-9c30-d41dbcfcf11b-kube-api-access-49rvg\") pod \"payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.431502 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.431354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001690c8-6863-414e-9c30-d41dbcfcf11b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.432059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.432040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001690c8-6863-414e-9c30-d41dbcfcf11b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.439315 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.439287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rvg\" (UniqueName: \"kubernetes.io/projected/001690c8-6863-414e-9c30-d41dbcfcf11b-kube-api-access-49rvg\") pod \"payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.516839 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.516728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:09.562248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.562205    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:09.575190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.575051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:09.796668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.796531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"]
Sep 17 23:18:09.812229 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:18:09.812198    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65eeba0b_33ed_4ae2_a820_dac3b240fff7.slice/crio-dc5aaee93394fb611f2e04865f9a4b8970b3b27030b530fc0114f9c7f5655543 WatchSource:0}: Error finding container dc5aaee93394fb611f2e04865f9a4b8970b3b27030b530fc0114f9c7f5655543: Status 404 returned error can't find the container with id dc5aaee93394fb611f2e04865f9a4b8970b3b27030b530fc0114f9c7f5655543
Sep 17 23:18:09.878264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:09.878241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"]
Sep 17 23:18:09.879812 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:18:09.879774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001690c8_6863_414e_9c30_d41dbcfcf11b.slice/crio-fa3ad7f8caba0ff1ebca5c2efd8fee35bdc17eedadf3c62d7fa3dc89d33e1d19 WatchSource:0}: Error finding container fa3ad7f8caba0ff1ebca5c2efd8fee35bdc17eedadf3c62d7fa3dc89d33e1d19: Status 404 returned error can't find the container with id fa3ad7f8caba0ff1ebca5c2efd8fee35bdc17eedadf3c62d7fa3dc89d33e1d19
Sep 17 23:18:10.091172 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.091067    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 17 23:18:10.091172 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.091141    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 17 23:18:10.295385 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.295342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:10.474491 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.474447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:10.512578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.512535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld" event={"ID":"65eeba0b-33ed-4ae2-a820-dac3b240fff7","Type":"ContainerStarted","Data":"1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8"}
Sep 17 23:18:10.512755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.512621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld" event={"ID":"65eeba0b-33ed-4ae2-a820-dac3b240fff7","Type":"ContainerStarted","Data":"dc5aaee93394fb611f2e04865f9a4b8970b3b27030b530fc0114f9c7f5655543"}
Sep 17 23:18:10.513686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.513649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:10.516036 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.516006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v" event={"ID":"001690c8-6863-414e-9c30-d41dbcfcf11b","Type":"ContainerStarted","Data":"ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420"}
Sep 17 23:18:10.516146 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.516073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v" event={"ID":"001690c8-6863-414e-9c30-d41dbcfcf11b","Type":"ContainerStarted","Data":"fa3ad7f8caba0ff1ebca5c2efd8fee35bdc17eedadf3c62d7fa3dc89d33e1d19"}
Sep 17 23:18:10.516409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.516390    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:10.532516 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.532466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld" podStartSLOduration=1.53245026 podStartE2EDuration="1.53245026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:10.531510852 +0000 UTC m=+700.961370669" watchObservedRunningTime="2026-09-17 23:18:10.53245026 +0000 UTC m=+700.962310078"
Sep 17 23:18:10.550209 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:10.550164    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v" podStartSLOduration=1.5501486450000002 podStartE2EDuration="1.550148645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:10.549401598 +0000 UTC m=+700.979261417" watchObservedRunningTime="2026-09-17 23:18:10.550148645 +0000 UTC m=+700.980008463"
Sep 17 23:18:11.095557 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:11.095522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:18:11.096071 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:11.096049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:18:11.096777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:11.096758    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:18:11.667041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:11.667000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:12.643724 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:12.643692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:13.318787 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:13.318750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:13.963681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:13.963640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:13.970337 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:13.970311    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"]
Sep 17 23:18:13.978852 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:13.978823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"
Sep 17 23:18:13.981459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:13.981436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:18:13.981829 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:13.981785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:18:13.982023 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:13.981999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"]
Sep 17 23:18:14.077053 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:14.077018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31f8eb2-e470-402c-baaa-ea50dac6f45d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh\" (UID: \"d31f8eb2-e470-402c-baaa-ea50dac6f45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"
Sep 17 23:18:14.177874 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:14.177835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31f8eb2-e470-402c-baaa-ea50dac6f45d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh\" (UID: \"d31f8eb2-e470-402c-baaa-ea50dac6f45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"
Sep 17 23:18:14.178470 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:14.178449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31f8eb2-e470-402c-baaa-ea50dac6f45d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh\" (UID: \"d31f8eb2-e470-402c-baaa-ea50dac6f45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"
Sep 17 23:18:14.292717 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:14.292624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"
Sep 17 23:18:14.564450 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:14.564423    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"]
Sep 17 23:18:14.566367 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:18:14.566339    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31f8eb2_e470_402c_baaa_ea50dac6f45d.slice/crio-e04911367a41b014a71ce9871c3448eb3d58d6ba39ca61756121cce5ca60ad05 WatchSource:0}: Error finding container e04911367a41b014a71ce9871c3448eb3d58d6ba39ca61756121cce5ca60ad05: Status 404 returned error can't find the container with id e04911367a41b014a71ce9871c3448eb3d58d6ba39ca61756121cce5ca60ad05
Sep 17 23:18:14.846606 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:14.846497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:15.485734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:15.485698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:15.543050 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:15.542946    2580 generic.go:356] "Generic (PLEG): container finished" podID="d31f8eb2-e470-402c-baaa-ea50dac6f45d" containerID="3596fc5259ef5f4d96bd5d147e45696596a65146ef69fe6560a052223b35605a" exitCode=0
Sep 17 23:18:15.543220 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:15.543044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh" event={"ID":"d31f8eb2-e470-402c-baaa-ea50dac6f45d","Type":"ContainerDied","Data":"3596fc5259ef5f4d96bd5d147e45696596a65146ef69fe6560a052223b35605a"}
Sep 17 23:18:15.543220 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:15.543095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh" event={"ID":"d31f8eb2-e470-402c-baaa-ea50dac6f45d","Type":"ContainerStarted","Data":"e04911367a41b014a71ce9871c3448eb3d58d6ba39ca61756121cce5ca60ad05"}
Sep 17 23:18:16.264886 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:16.264855    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:16.683052 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:16.683027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"
Sep 17 23:18:16.803903 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:16.803819    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31f8eb2-e470-402c-baaa-ea50dac6f45d-maas-api-service-ca\") pod \"d31f8eb2-e470-402c-baaa-ea50dac6f45d\" (UID: \"d31f8eb2-e470-402c-baaa-ea50dac6f45d\") "
Sep 17 23:18:16.804205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:16.804182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31f8eb2-e470-402c-baaa-ea50dac6f45d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d31f8eb2-e470-402c-baaa-ea50dac6f45d" (UID: "d31f8eb2-e470-402c-baaa-ea50dac6f45d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:18:16.904634 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:16.904593    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31f8eb2-e470-402c-baaa-ea50dac6f45d-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:18:17.076712 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:17.076635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:17.555551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:17.555515    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"
Sep 17 23:18:17.555551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:17.555519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh" event={"ID":"d31f8eb2-e470-402c-baaa-ea50dac6f45d","Type":"ContainerDied","Data":"e04911367a41b014a71ce9871c3448eb3d58d6ba39ca61756121cce5ca60ad05"}
Sep 17 23:18:17.555764 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:17.555582    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e04911367a41b014a71ce9871c3448eb3d58d6ba39ca61756121cce5ca60ad05"
Sep 17 23:18:17.612043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:17.612005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:17.714042 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:17.714008    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:17.834910 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:17.834823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:18.127422 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:18.127322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:19.014012 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.013977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:19.537425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.537391    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"]
Sep 17 23:18:19.537821 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.537783    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31f8eb2-e470-402c-baaa-ea50dac6f45d" containerName="revoke-keys"
Sep 17 23:18:19.537821 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.537812    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31f8eb2-e470-402c-baaa-ea50dac6f45d" containerName="revoke-keys"
Sep 17 23:18:19.537997 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.537892    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31f8eb2-e470-402c-baaa-ea50dac6f45d" containerName="revoke-keys"
Sep 17 23:18:19.541350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.541332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.543682 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.543664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ab633\"/\"default-dockercfg-b9f5k\""
Sep 17 23:18:19.544779 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.544758    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ab633\"/\"openshift-service-ca.crt\""
Sep 17 23:18:19.544779 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.544771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ab633\"/\"simulator-w4-7ab633-kserve-self-signed-certs\""
Sep 17 23:18:19.544943 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.544811    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ab633\"/\"kube-root-ca.crt\""
Sep 17 23:18:19.553296 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.553267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"]
Sep 17 23:18:19.631950 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.631919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-model-cache\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.632092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.631969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wx5\" (UniqueName: \"kubernetes.io/projected/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-kube-api-access-k6wx5\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.632160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.632091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tmp-dir\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.632160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.632138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-home\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.632226 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.632207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-dshm\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.632268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.632241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tls-certs\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733301 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-model-cache\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wx5\" (UniqueName: \"kubernetes.io/projected/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-kube-api-access-k6wx5\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tmp-dir\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-home\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-dshm\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tls-certs\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733766 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-model-cache\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733873 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tmp-dir\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.733873 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.733843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-home\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.735854 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.735834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-dshm\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.735960 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.735943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tls-certs\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.758960 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.758935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wx5\" (UniqueName: \"kubernetes.io/projected/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-kube-api-access-k6wx5\") pod \"simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") " pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.850509 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.850418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:19.890732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:19.890701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:20.138609 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:20.138582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"]
Sep 17 23:18:20.141094 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:18:20.141063    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c8df38b_006a_4b5d_9b52_2a5a66d2cabf.slice/crio-41b53b5f44138c18902503c1ed48c570dd4ae80ed1f2f8889102f914aa049058 WatchSource:0}: Error finding container 41b53b5f44138c18902503c1ed48c570dd4ae80ed1f2f8889102f914aa049058: Status 404 returned error can't find the container with id 41b53b5f44138c18902503c1ed48c570dd4ae80ed1f2f8889102f914aa049058
Sep 17 23:18:20.166478 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:20.166453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:20.572618 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:20.572523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz" event={"ID":"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf","Type":"ContainerStarted","Data":"e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2"}
Sep 17 23:18:20.572618 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:20.572570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz" event={"ID":"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf","Type":"ContainerStarted","Data":"41b53b5f44138c18902503c1ed48c570dd4ae80ed1f2f8889102f914aa049058"}
Sep 17 23:18:20.603903 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:20.603853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz" podStartSLOduration=1.6038383189999998 podStartE2EDuration="1.603838319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:20.601963254 +0000 UTC m=+711.031823072" watchObservedRunningTime="2026-09-17 23:18:20.603838319 +0000 UTC m=+711.033698137"
Sep 17 23:18:21.191782 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:21.191734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:21.523458 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:21.523373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:18:22.528969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:22.528939    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:18:23.763320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:23.763287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:23.819318 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:23.819284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:23.909463 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:23.909430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:23.995963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:23.995897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.111712 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.111609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.508945 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.508916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.563741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.563708    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"]
Sep 17 23:18:24.563983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.563945    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j" podUID="914bdc04-21ad-43fa-80c3-87a4d179c4fc" containerName="payload-pre-processing" containerID="cri-o://2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935" gracePeriod=30
Sep 17 23:18:24.594009 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.593972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.668504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.668468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.757876 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.757844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"]
Sep 17 23:18:24.760958 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.760880    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10a7edb-pr8zh"]
Sep 17 23:18:24.763602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.763575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.817983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.817961    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:18:24.832938 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.832911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.883515 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.883473    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvnfc\" (UniqueName: \"kubernetes.io/projected/914bdc04-21ad-43fa-80c3-87a4d179c4fc-kube-api-access-xvnfc\") pod \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") "
Sep 17 23:18:24.883708 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.883527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914bdc04-21ad-43fa-80c3-87a4d179c4fc-plugins-config-volume\") pod \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\" (UID: \"914bdc04-21ad-43fa-80c3-87a4d179c4fc\") "
Sep 17 23:18:24.883931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.883901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914bdc04-21ad-43fa-80c3-87a4d179c4fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "914bdc04-21ad-43fa-80c3-87a4d179c4fc" (UID: "914bdc04-21ad-43fa-80c3-87a4d179c4fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:18:24.886955 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.886926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914bdc04-21ad-43fa-80c3-87a4d179c4fc-kube-api-access-xvnfc" (OuterVolumeSpecName: "kube-api-access-xvnfc") pod "914bdc04-21ad-43fa-80c3-87a4d179c4fc" (UID: "914bdc04-21ad-43fa-80c3-87a4d179c4fc"). InnerVolumeSpecName "kube-api-access-xvnfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:18:24.975270 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.975233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:24.984541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.984511    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnfc\" (UniqueName: \"kubernetes.io/projected/914bdc04-21ad-43fa-80c3-87a4d179c4fc-kube-api-access-xvnfc\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:18:24.984541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:24.984540    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914bdc04-21ad-43fa-80c3-87a4d179c4fc-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:18:25.258046 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.258011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:25.383406 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.383373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:25.463064 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.463032    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:25.591520 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.591429    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:25.609404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.609372    2580 generic.go:356] "Generic (PLEG): container finished" podID="914bdc04-21ad-43fa-80c3-87a4d179c4fc" containerID="2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935" exitCode=0
Sep 17 23:18:25.609581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.609407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j" event={"ID":"914bdc04-21ad-43fa-80c3-87a4d179c4fc","Type":"ContainerDied","Data":"2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935"}
Sep 17 23:18:25.609581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.609445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j" event={"ID":"914bdc04-21ad-43fa-80c3-87a4d179c4fc","Type":"ContainerDied","Data":"891bf183dff88be665d8e414af0ebc94236f6bff310df4107a66a99cc0ebb05c"}
Sep 17 23:18:25.609581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.609454    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"
Sep 17 23:18:25.609581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.609466    2580 scope.go:117] "RemoveContainer" containerID="2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935"
Sep 17 23:18:25.623062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.623038    2580 scope.go:117] "RemoveContainer" containerID="2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935"
Sep 17 23:18:25.624194 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:18:25.624163    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935\": container with ID starting with 2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935 not found: ID does not exist" containerID="2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935"
Sep 17 23:18:25.624365 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.624205    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935"} err="failed to get container status \"2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935\": rpc error: code = NotFound desc = could not find container \"2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935\": container with ID starting with 2333932e4c8b220879cd1ab567d08ca718a5e05c151381afcd1dce7305b60935 not found: ID does not exist"
Sep 17 23:18:25.634479 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.634456    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"]
Sep 17 23:18:25.638246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.638219    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f10a7edb-657fc869b6-4972j"]
Sep 17 23:18:25.903362 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:25.903330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:26.108141 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:26.108110    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="914bdc04-21ad-43fa-80c3-87a4d179c4fc" path="/var/lib/kubelet/pods/914bdc04-21ad-43fa-80c3-87a4d179c4fc/volumes"
Sep 17 23:18:26.108452 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:26.108440    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d31f8eb2-e470-402c-baaa-ea50dac6f45d" path="/var/lib/kubelet/pods/d31f8eb2-e470-402c-baaa-ea50dac6f45d/volumes"
Sep 17 23:18:26.272677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:26.272596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:26.669113 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:26.669081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:27.048581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:27.048496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:28.051416 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:28.051381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:28.891398 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:28.891361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:28.994911 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:28.994881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:29.094746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:29.094719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:29.835777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:29.835740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:29.851329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:29.851304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:29.851589 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:29.851574    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:29.863664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:29.863643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:30.277415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:30.277377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:30.340291 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:30.340257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:30.418383 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:30.418349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:30.605520 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:30.605435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:30.647045 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:30.647009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:18:31.358063 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.358037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:31.459971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.459941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:31.514728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.514692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:31.844766 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.844734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:31.920511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.920477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:31.990510 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.990472    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"]
Sep 17 23:18:31.990898 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.990885    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914bdc04-21ad-43fa-80c3-87a4d179c4fc" containerName="payload-pre-processing"
Sep 17 23:18:31.990956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.990899    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="914bdc04-21ad-43fa-80c3-87a4d179c4fc" containerName="payload-pre-processing"
Sep 17 23:18:31.990994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.990961    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="914bdc04-21ad-43fa-80c3-87a4d179c4fc" containerName="payload-pre-processing"
Sep 17 23:18:31.995588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.995565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:31.998061 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.998043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b75118e5-dockercfg-24xfm\""
Sep 17 23:18:31.998162 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:31.998147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b75118e5\""
Sep 17 23:18:32.005051 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.005027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"]
Sep 17 23:18:32.154295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.154258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k642\" (UniqueName: \"kubernetes.io/projected/5db133d1-cc70-4873-8c02-35cf2468b956-kube-api-access-4k642\") pod \"payload-processing-e2e-migrate-b75118e5-548956f75d-944vs\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") " pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.154295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.154298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db133d1-cc70-4873-8c02-35cf2468b956-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b75118e5-548956f75d-944vs\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") " pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.255790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.255744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k642\" (UniqueName: \"kubernetes.io/projected/5db133d1-cc70-4873-8c02-35cf2468b956-kube-api-access-4k642\") pod \"payload-processing-e2e-migrate-b75118e5-548956f75d-944vs\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") " pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.256027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.255829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db133d1-cc70-4873-8c02-35cf2468b956-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b75118e5-548956f75d-944vs\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") " pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.256511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.256489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db133d1-cc70-4873-8c02-35cf2468b956-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b75118e5-548956f75d-944vs\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") " pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.263348 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.263321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k642\" (UniqueName: \"kubernetes.io/projected/5db133d1-cc70-4873-8c02-35cf2468b956-kube-api-access-4k642\") pod \"payload-processing-e2e-migrate-b75118e5-548956f75d-944vs\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") " pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.307421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.307379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.544753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.544666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:32.581247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.581221    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"]
Sep 17 23:18:32.583553 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:18:32.583524    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db133d1_cc70_4873_8c02_35cf2468b956.slice/crio-1f00cce2761f1a5d104d0e52cb545e3feda13f08b900a5002eb0c5b648f85cb2 WatchSource:0}: Error finding container 1f00cce2761f1a5d104d0e52cb545e3feda13f08b900a5002eb0c5b648f85cb2: Status 404 returned error can't find the container with id 1f00cce2761f1a5d104d0e52cb545e3feda13f08b900a5002eb0c5b648f85cb2
Sep 17 23:18:32.647955 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.647925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs" event={"ID":"5db133d1-cc70-4873-8c02-35cf2468b956","Type":"ContainerStarted","Data":"d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029"}
Sep 17 23:18:32.648084 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.647987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs" event={"ID":"5db133d1-cc70-4873-8c02-35cf2468b956","Type":"ContainerStarted","Data":"1f00cce2761f1a5d104d0e52cb545e3feda13f08b900a5002eb0c5b648f85cb2"}
Sep 17 23:18:32.648215 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.648195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:32.665403 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:32.665356    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs" podStartSLOduration=1.665340714 podStartE2EDuration="1.665340714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:32.663582907 +0000 UTC m=+723.093442724" watchObservedRunningTime="2026-09-17 23:18:32.665340714 +0000 UTC m=+723.095200575"
Sep 17 23:18:33.291689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:33.291572    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:33.946876 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:33.946819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.748745 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:34.748710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.662366 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:43.662333    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:18:50.954012 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:50.953973    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"]
Sep 17 23:18:50.958906 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:50.958885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:50.962369 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:50.962340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0b4d93\"/\"default-dockercfg-fzhhg\""
Sep 17 23:18:50.962522 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:50.962400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0b4d93\"/\"kube-root-ca.crt\""
Sep 17 23:18:50.962522 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:50.962348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0b4d93\"/\"openshift-service-ca.crt\""
Sep 17 23:18:50.962522 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:50.962346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0b4d93\"/\"simulator-w0-0b4d93-kserve-self-signed-certs\""
Sep 17 23:18:50.967328 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:50.967309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"]
Sep 17 23:18:51.036722 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.036685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwcxd\" (UniqueName: \"kubernetes.io/projected/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-kube-api-access-cwcxd\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.036932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.036732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tmp-dir\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.036932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.036763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-home\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.036932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.036794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-model-cache\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.036932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.036887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tls-certs\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.037182 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.037052    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-dshm\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.069356 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.069321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:51.138160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-dshm\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcxd\" (UniqueName: \"kubernetes.io/projected/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-kube-api-access-cwcxd\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tmp-dir\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-home\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-model-cache\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tls-certs\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138685 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-home\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138895 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tmp-dir\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.138973 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.138725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-model-cache\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.140751 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.140724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-dshm\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.140955 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.140938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tls-certs\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.147329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.147308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcxd\" (UniqueName: \"kubernetes.io/projected/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-kube-api-access-cwcxd\") pod \"simulator-w0-0b4d93-kserve-96f48887-h9bl8\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") " pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.269410 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.269303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:18:51.543685 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.543602    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"]
Sep 17 23:18:51.547164 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:18:51.547133    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ac41bd9_8645_4d0f_b023_5c7fd42e71e0.slice/crio-171f602298904a99d62f4a6036da812dded7c644b0155f17845fb1cf68a95d3f WatchSource:0}: Error finding container 171f602298904a99d62f4a6036da812dded7c644b0155f17845fb1cf68a95d3f: Status 404 returned error can't find the container with id 171f602298904a99d62f4a6036da812dded7c644b0155f17845fb1cf68a95d3f
Sep 17 23:18:51.763481 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.763440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8" event={"ID":"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0","Type":"ContainerStarted","Data":"f8de253b99580e8009e16df59d650fe55dae494cd906c50b70a6f9a67125ffb1"}
Sep 17 23:18:51.763481 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.763487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8" event={"ID":"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0","Type":"ContainerStarted","Data":"171f602298904a99d62f4a6036da812dded7c644b0155f17845fb1cf68a95d3f"}
Sep 17 23:18:51.782305 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:51.782250    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8" podStartSLOduration=1.7822345739999998 podStartE2EDuration="1.782234574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:51.780179664 +0000 UTC m=+742.210039482" watchObservedRunningTime="2026-09-17 23:18:51.782234574 +0000 UTC m=+742.212094383"
Sep 17 23:18:53.678635 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:53.678595    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:54.486100 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:54.486068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:55.349429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:55.349389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:56.049142 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:56.049102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:56.976688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:56.976650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:57.728635 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:57.728605    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:58.658969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:58.658920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:59.520357 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:18:59.520325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:01.269637 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:01.269592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:19:01.270056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:01.269659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:19:01.282533 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:01.282505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:19:01.832419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:01.832389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:19:08.143364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:08.143333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:10.128187 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:10.128154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:10.852573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:10.852529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:10.878118 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:10.878082    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"]
Sep 17 23:19:10.878349 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:10.878310    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs" podUID="5db133d1-cc70-4873-8c02-35cf2468b956" containerName="payload-processing" containerID="cri-o://d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029" gracePeriod=30
Sep 17 23:19:10.952451 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:10.952419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:11.041212 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.041181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:11.114296 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.114208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:11.146494 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.146467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:19:11.197654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.197620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:11.241703 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.241663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db133d1-cc70-4873-8c02-35cf2468b956-plugins-config-volume\") pod \"5db133d1-cc70-4873-8c02-35cf2468b956\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") "
Sep 17 23:19:11.241939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.241717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k642\" (UniqueName: \"kubernetes.io/projected/5db133d1-cc70-4873-8c02-35cf2468b956-kube-api-access-4k642\") pod \"5db133d1-cc70-4873-8c02-35cf2468b956\" (UID: \"5db133d1-cc70-4873-8c02-35cf2468b956\") "
Sep 17 23:19:11.242086 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.242064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db133d1-cc70-4873-8c02-35cf2468b956-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5db133d1-cc70-4873-8c02-35cf2468b956" (UID: "5db133d1-cc70-4873-8c02-35cf2468b956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:11.242177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.242165    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db133d1-cc70-4873-8c02-35cf2468b956-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:19:11.243959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.243940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db133d1-cc70-4873-8c02-35cf2468b956-kube-api-access-4k642" (OuterVolumeSpecName: "kube-api-access-4k642") pod "5db133d1-cc70-4873-8c02-35cf2468b956" (UID: "5db133d1-cc70-4873-8c02-35cf2468b956"). InnerVolumeSpecName "kube-api-access-4k642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:11.343143 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.343104    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k642\" (UniqueName: \"kubernetes.io/projected/5db133d1-cc70-4873-8c02-35cf2468b956-kube-api-access-4k642\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:19:11.456734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.456700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:11.629133 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.629102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:11.875266 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.875174    2580 generic.go:356] "Generic (PLEG): container finished" podID="5db133d1-cc70-4873-8c02-35cf2468b956" containerID="d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029" exitCode=0
Sep 17 23:19:11.875266 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.875222    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"
Sep 17 23:19:11.875514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.875261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs" event={"ID":"5db133d1-cc70-4873-8c02-35cf2468b956","Type":"ContainerDied","Data":"d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029"}
Sep 17 23:19:11.875514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.875322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs" event={"ID":"5db133d1-cc70-4873-8c02-35cf2468b956","Type":"ContainerDied","Data":"1f00cce2761f1a5d104d0e52cb545e3feda13f08b900a5002eb0c5b648f85cb2"}
Sep 17 23:19:11.875514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.875354    2580 scope.go:117] "RemoveContainer" containerID="d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029"
Sep 17 23:19:11.885602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.885397    2580 scope.go:117] "RemoveContainer" containerID="d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029"
Sep 17 23:19:11.885975 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:19:11.885945    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029\": container with ID starting with d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029 not found: ID does not exist" containerID="d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029"
Sep 17 23:19:11.886111 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.886072    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029"} err="failed to get container status \"d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029\": rpc error: code = NotFound desc = could not find container \"d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029\": container with ID starting with d8be617cc3c28ec93e0690be12dd6b3b6e5543a8382cddbe22b8a5eadcfeb029 not found: ID does not exist"
Sep 17 23:19:11.902577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.902548    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"]
Sep 17 23:19:11.916467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.916437    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b75118e5-548956f75d-944vs"]
Sep 17 23:19:11.956953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:11.956923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:12.081303 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:12.081272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:12.109288 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:12.109258    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db133d1-cc70-4873-8c02-35cf2468b956" path="/var/lib/kubelet/pods/5db133d1-cc70-4873-8c02-35cf2468b956/volumes"
Sep 17 23:19:12.428196 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:12.428164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:13.343315 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:13.343279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:14.165762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:14.165720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:14.265603 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:14.265568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:15.163507 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:15.163477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:16.141716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:16.141685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:16.854108 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:16.854074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:16.953597 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:16.953563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:17.110253 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.110169    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:17.504501 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.504469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:17.603091 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.603058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:17.681479 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.681438    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"]
Sep 17 23:19:17.681973 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.681946    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db133d1-cc70-4873-8c02-35cf2468b956" containerName="payload-processing"
Sep 17 23:19:17.681973 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.681973    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db133d1-cc70-4873-8c02-35cf2468b956" containerName="payload-processing"
Sep 17 23:19:17.682170 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.682086    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db133d1-cc70-4873-8c02-35cf2468b956" containerName="payload-processing"
Sep 17 23:19:17.693726 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.693700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:17.696465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.696442    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-92cb6c75-dockercfg-xvssz\""
Sep 17 23:19:17.696658 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.696483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-92cb6c75\""
Sep 17 23:19:17.697795 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.697775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"]
Sep 17 23:19:17.705544 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.705522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235c3d05-f364-4981-81cd-7514ad0e1d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:17.705654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.705558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cb85\" (UniqueName: \"kubernetes.io/projected/235c3d05-f364-4981-81cd-7514ad0e1d76-kube-api-access-8cb85\") pod \"payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:17.806609 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.806518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cb85\" (UniqueName: \"kubernetes.io/projected/235c3d05-f364-4981-81cd-7514ad0e1d76-kube-api-access-8cb85\") pod \"payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:17.806767 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.806679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235c3d05-f364-4981-81cd-7514ad0e1d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:17.807408 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.807389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235c3d05-f364-4981-81cd-7514ad0e1d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:17.817194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:17.817172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cb85\" (UniqueName: \"kubernetes.io/projected/235c3d05-f364-4981-81cd-7514ad0e1d76-kube-api-access-8cb85\") pod \"payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:18.006208 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.006174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:18.066573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.066536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:18.286180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.286154    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"]
Sep 17 23:19:18.288581 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:19:18.288555    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235c3d05_f364_4981_81cd_7514ad0e1d76.slice/crio-71c972fd48b524dc85b75ac96e699625410f3d5269810eabf91b8e7e6d7507a1 WatchSource:0}: Error finding container 71c972fd48b524dc85b75ac96e699625410f3d5269810eabf91b8e7e6d7507a1: Status 404 returned error can't find the container with id 71c972fd48b524dc85b75ac96e699625410f3d5269810eabf91b8e7e6d7507a1
Sep 17 23:19:18.918882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.918849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t" event={"ID":"235c3d05-f364-4981-81cd-7514ad0e1d76","Type":"ContainerStarted","Data":"c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33"}
Sep 17 23:19:18.919378 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.918904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t" event={"ID":"235c3d05-f364-4981-81cd-7514ad0e1d76","Type":"ContainerStarted","Data":"71c972fd48b524dc85b75ac96e699625410f3d5269810eabf91b8e7e6d7507a1"}
Sep 17 23:19:18.919378 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.918932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:18.924682 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.924650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:18.940688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:18.940633    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t" podStartSLOduration=1.940616972 podStartE2EDuration="1.940616972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:18.940111276 +0000 UTC m=+769.369971099" watchObservedRunningTime="2026-09-17 23:19:18.940616972 +0000 UTC m=+769.370476790"
Sep 17 23:19:19.625440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:19.625404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:20.561716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:20.561679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:29.926709 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:29.926676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:37.745876 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.745837    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"]
Sep 17 23:19:37.750564 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.750539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:37.753932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.753903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7aa654\"/\"test-model-7aa654-kserve-self-signed-certs\""
Sep 17 23:19:37.754183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.753903    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7aa654\"/\"openshift-service-ca.crt\""
Sep 17 23:19:37.754183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.753907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7aa654\"/\"default-dockercfg-7t25h\""
Sep 17 23:19:37.754183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.753909    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7aa654\"/\"kube-root-ca.crt\""
Sep 17 23:19:37.762239 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.762211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"]
Sep 17 23:19:37.781332 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.781283    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:37.906201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.906160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f279n\" (UniqueName: \"kubernetes.io/projected/2a3c8049-f61e-499b-8bc5-550a56268f12-kube-api-access-f279n\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:37.906396 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.906210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-tmp-dir\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:37.906396 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.906267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-model-cache\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:37.906396 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.906288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3c8049-f61e-499b-8bc5-550a56268f12-tls-certs\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:37.906396 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.906337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-dshm\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:37.906396 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:37.906362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-home\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007450 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f279n\" (UniqueName: \"kubernetes.io/projected/2a3c8049-f61e-499b-8bc5-550a56268f12-kube-api-access-f279n\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007450 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-tmp-dir\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007450 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-model-cache\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3c8049-f61e-499b-8bc5-550a56268f12-tls-certs\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-dshm\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-home\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007928 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-tmp-dir\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.007987 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.007973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-home\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.008046 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.008010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-model-cache\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.010051 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.010023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-dshm\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.010369 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.010348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3c8049-f61e-499b-8bc5-550a56268f12-tls-certs\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.015135 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.015113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f279n\" (UniqueName: \"kubernetes.io/projected/2a3c8049-f61e-499b-8bc5-550a56268f12-kube-api-access-f279n\") pod \"test-model-7aa654-kserve-55cccdd465-2vkff\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") " pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.062091 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.062052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:38.338349 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:38.338321    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"]
Sep 17 23:19:38.340863 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:19:38.340832    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a3c8049_f61e_499b_8bc5_550a56268f12.slice/crio-d92a78419f4cf8f5fa96049734579130537bdc7815938a6997dac772a43b08f3 WatchSource:0}: Error finding container d92a78419f4cf8f5fa96049734579130537bdc7815938a6997dac772a43b08f3: Status 404 returned error can't find the container with id d92a78419f4cf8f5fa96049734579130537bdc7815938a6997dac772a43b08f3
Sep 17 23:19:39.029232 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:39.029193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff" event={"ID":"2a3c8049-f61e-499b-8bc5-550a56268f12","Type":"ContainerStarted","Data":"d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650"}
Sep 17 23:19:39.029690 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:39.029241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff" event={"ID":"2a3c8049-f61e-499b-8bc5-550a56268f12","Type":"ContainerStarted","Data":"d92a78419f4cf8f5fa96049734579130537bdc7815938a6997dac772a43b08f3"}
Sep 17 23:19:39.046770 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:39.046714    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff" podStartSLOduration=2.046697539 podStartE2EDuration="2.046697539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:39.044968729 +0000 UTC m=+789.474828547" watchObservedRunningTime="2026-09-17 23:19:39.046697539 +0000 UTC m=+789.476557358"
Sep 17 23:19:39.163427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:39.163394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:40.032047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:40.032012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:40.844315 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:40.844277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:41.500774 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:41.500740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:42.220427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:42.220394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:43.058588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.058555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:43.235283 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.235240    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"]
Sep 17 23:19:43.239066 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.239037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"
Sep 17 23:19:43.241441 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.241421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:19:43.241558 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.241424    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:19:43.247885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.247859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"]
Sep 17 23:19:43.360335 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.360242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca6d3b0-7e6a-4be1-9518-5b42be61c575-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6114ed-tpnf5\" (UID: \"eca6d3b0-7e6a-4be1-9518-5b42be61c575\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"
Sep 17 23:19:43.467892 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.467847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca6d3b0-7e6a-4be1-9518-5b42be61c575-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6114ed-tpnf5\" (UID: \"eca6d3b0-7e6a-4be1-9518-5b42be61c575\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"
Sep 17 23:19:43.468970 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.468946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca6d3b0-7e6a-4be1-9518-5b42be61c575-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6114ed-tpnf5\" (UID: \"eca6d3b0-7e6a-4be1-9518-5b42be61c575\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"
Sep 17 23:19:43.552568 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.552539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"
Sep 17 23:19:43.825895 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:43.825868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"]
Sep 17 23:19:43.828373 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:19:43.828346    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca6d3b0_7e6a_4be1_9518_5b42be61c575.slice/crio-7c56350b669af79a9a7feeeda673f85c98931dac54d9a6228e0be9c3cf7f7759 WatchSource:0}: Error finding container 7c56350b669af79a9a7feeeda673f85c98931dac54d9a6228e0be9c3cf7f7759: Status 404 returned error can't find the container with id 7c56350b669af79a9a7feeeda673f85c98931dac54d9a6228e0be9c3cf7f7759
Sep 17 23:19:44.056605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:44.056575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5" event={"ID":"eca6d3b0-7e6a-4be1-9518-5b42be61c575","Type":"ContainerStarted","Data":"7c56350b669af79a9a7feeeda673f85c98931dac54d9a6228e0be9c3cf7f7759"}
Sep 17 23:19:44.059708 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:44.059673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:44.862444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:44.862361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:45.009194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:45.009161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:45.064038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:45.063999    2580 generic.go:356] "Generic (PLEG): container finished" podID="eca6d3b0-7e6a-4be1-9518-5b42be61c575" containerID="e75af2c1ae7d9c1a6255b5884f91b4634a4f114d9fad8b59479f01a51dc19ce2" exitCode=0
Sep 17 23:19:45.064414 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:45.064093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5" event={"ID":"eca6d3b0-7e6a-4be1-9518-5b42be61c575","Type":"ContainerDied","Data":"e75af2c1ae7d9c1a6255b5884f91b4634a4f114d9fad8b59479f01a51dc19ce2"}
Sep 17 23:19:45.574331 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:45.574300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:46.212379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:46.212349    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"
Sep 17 23:19:46.339649 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:46.339563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:46.398027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:46.397994    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca6d3b0-7e6a-4be1-9518-5b42be61c575-maas-api-service-ca\") pod \"eca6d3b0-7e6a-4be1-9518-5b42be61c575\" (UID: \"eca6d3b0-7e6a-4be1-9518-5b42be61c575\") "
Sep 17 23:19:46.398341 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:46.398318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca6d3b0-7e6a-4be1-9518-5b42be61c575-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eca6d3b0-7e6a-4be1-9518-5b42be61c575" (UID: "eca6d3b0-7e6a-4be1-9518-5b42be61c575"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:46.498757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:46.498725    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca6d3b0-7e6a-4be1-9518-5b42be61c575-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:19:46.971571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:46.971537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:47.076449 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:47.076417    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"
Sep 17 23:19:47.076449 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:47.076445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5" event={"ID":"eca6d3b0-7e6a-4be1-9518-5b42be61c575","Type":"ContainerDied","Data":"7c56350b669af79a9a7feeeda673f85c98931dac54d9a6228e0be9c3cf7f7759"}
Sep 17 23:19:47.076651 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:47.076491    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c56350b669af79a9a7feeeda673f85c98931dac54d9a6228e0be9c3cf7f7759"
Sep 17 23:19:47.649433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:47.649383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:47.997225 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:47.997144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:48.062816 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:48.062766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:48.063004 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:48.062850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:48.077716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:48.077683    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:48.096011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:48.095984    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:19:48.449147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:48.449109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:49.018612 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.018579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:49.049732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.049688    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"]
Sep 17 23:19:49.050029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.049996    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t" podUID="235c3d05-f364-4981-81cd-7514ad0e1d76" containerName="payload-pre-processing" containerID="cri-o://c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33" gracePeriod=30
Sep 17 23:19:49.118916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.118879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:49.202012 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.201959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:49.264851 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.264823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:49.299570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.299546    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:49.431510 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.431467    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cb85\" (UniqueName: \"kubernetes.io/projected/235c3d05-f364-4981-81cd-7514ad0e1d76-kube-api-access-8cb85\") pod \"235c3d05-f364-4981-81cd-7514ad0e1d76\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") "
Sep 17 23:19:49.431713 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.431561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235c3d05-f364-4981-81cd-7514ad0e1d76-plugins-config-volume\") pod \"235c3d05-f364-4981-81cd-7514ad0e1d76\" (UID: \"235c3d05-f364-4981-81cd-7514ad0e1d76\") "
Sep 17 23:19:49.431980 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.431949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235c3d05-f364-4981-81cd-7514ad0e1d76-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "235c3d05-f364-4981-81cd-7514ad0e1d76" (UID: "235c3d05-f364-4981-81cd-7514ad0e1d76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:49.433865 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.433846    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235c3d05-f364-4981-81cd-7514ad0e1d76-kube-api-access-8cb85" (OuterVolumeSpecName: "kube-api-access-8cb85") pod "235c3d05-f364-4981-81cd-7514ad0e1d76" (UID: "235c3d05-f364-4981-81cd-7514ad0e1d76"). InnerVolumeSpecName "kube-api-access-8cb85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:49.533017 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.532938    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cb85\" (UniqueName: \"kubernetes.io/projected/235c3d05-f364-4981-81cd-7514ad0e1d76-kube-api-access-8cb85\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:19:49.533017 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.532968    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235c3d05-f364-4981-81cd-7514ad0e1d76-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:19:49.538001 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:49.537973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:50.098634 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.098600    2580 generic.go:356] "Generic (PLEG): container finished" podID="235c3d05-f364-4981-81cd-7514ad0e1d76" containerID="c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33" exitCode=0
Sep 17 23:19:50.099222 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.098658    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"
Sep 17 23:19:50.099222 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.098689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t" event={"ID":"235c3d05-f364-4981-81cd-7514ad0e1d76","Type":"ContainerDied","Data":"c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33"}
Sep 17 23:19:50.099222 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.098740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t" event={"ID":"235c3d05-f364-4981-81cd-7514ad0e1d76","Type":"ContainerDied","Data":"71c972fd48b524dc85b75ac96e699625410f3d5269810eabf91b8e7e6d7507a1"}
Sep 17 23:19:50.099222 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.098764    2580 scope.go:117] "RemoveContainer" containerID="c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33"
Sep 17 23:19:50.109242 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.109222    2580 scope.go:117] "RemoveContainer" containerID="c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33"
Sep 17 23:19:50.109505 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:19:50.109486    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33\": container with ID starting with c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33 not found: ID does not exist" containerID="c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33"
Sep 17 23:19:50.109563 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.109514    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33"} err="failed to get container status \"c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33\": rpc error: code = NotFound desc = could not find container \"c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33\": container with ID starting with c48984f39752611f22788ef8917921a7bec0b7b55384d55b3ddd4afd99b7ed33 not found: ID does not exist"
Sep 17 23:19:50.121571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.121544    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"]
Sep 17 23:19:50.126395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.126374    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92cb6c75-75c6957c46-9jg6t"]
Sep 17 23:19:50.339374 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.339342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:50.414740 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.414710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:50.803086 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:50.802985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:51.092238 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:51.092152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:51.178256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:51.178226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:51.434955 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:51.434923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:51.961162 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:51.961128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:52.108921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:52.108870    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="235c3d05-f364-4981-81cd-7514ad0e1d76" path="/var/lib/kubelet/pods/235c3d05-f364-4981-81cd-7514ad0e1d76/volumes"
Sep 17 23:19:52.842523 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:52.842490    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:53.291219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:53.291183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:53.387488 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:53.387444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:53.554096 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:53.554010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:54.431168 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:54.431138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:55.139732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:55.139702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:55.801109 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:55.801073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:56.561977 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:56.561940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.171325 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.171278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.199409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.199373    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"]
Sep 17 23:19:57.199657 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.199629    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8" podUID="c4437ed2-2c09-4b1f-b712-05c2f824f24f" containerName="payload-pre-processing" containerID="cri-o://1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e" gracePeriod=30
Sep 17 23:19:57.271771 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.271739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.372844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.372790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.395107 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.395075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"]
Sep 17 23:19:57.399707 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.399681    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6114ed-tpnf5"]
Sep 17 23:19:57.464267 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.464241    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:19:57.480604 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.480574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.512921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.512891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4437ed2-2c09-4b1f-b712-05c2f824f24f-plugins-config-volume\") pod \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") "
Sep 17 23:19:57.513084 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.512947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bthvl\" (UniqueName: \"kubernetes.io/projected/c4437ed2-2c09-4b1f-b712-05c2f824f24f-kube-api-access-bthvl\") pod \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\" (UID: \"c4437ed2-2c09-4b1f-b712-05c2f824f24f\") "
Sep 17 23:19:57.513347 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.513324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4437ed2-2c09-4b1f-b712-05c2f824f24f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4437ed2-2c09-4b1f-b712-05c2f824f24f" (UID: "c4437ed2-2c09-4b1f-b712-05c2f824f24f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:57.515205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.515184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4437ed2-2c09-4b1f-b712-05c2f824f24f-kube-api-access-bthvl" (OuterVolumeSpecName: "kube-api-access-bthvl") pod "c4437ed2-2c09-4b1f-b712-05c2f824f24f" (UID: "c4437ed2-2c09-4b1f-b712-05c2f824f24f"). InnerVolumeSpecName "kube-api-access-bthvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:57.548254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.548226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.614387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.614356    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4437ed2-2c09-4b1f-b712-05c2f824f24f-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:19:57.614387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.614381    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bthvl\" (UniqueName: \"kubernetes.io/projected/c4437ed2-2c09-4b1f-b712-05c2f824f24f-kube-api-access-bthvl\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:19:57.676073 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.676033    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.776194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.776090    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:57.973959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:57.973925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:58.072679 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.072580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:58.110216 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.110178    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca6d3b0-7e6a-4be1-9518-5b42be61c575" path="/var/lib/kubelet/pods/eca6d3b0-7e6a-4be1-9518-5b42be61c575/volumes"
Sep 17 23:19:58.144784 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.144753    2580 generic.go:356] "Generic (PLEG): container finished" podID="c4437ed2-2c09-4b1f-b712-05c2f824f24f" containerID="1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e" exitCode=0
Sep 17 23:19:58.144959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.144833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8" event={"ID":"c4437ed2-2c09-4b1f-b712-05c2f824f24f","Type":"ContainerDied","Data":"1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e"}
Sep 17 23:19:58.144959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.144907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8" event={"ID":"c4437ed2-2c09-4b1f-b712-05c2f824f24f","Type":"ContainerDied","Data":"543badb03090055f4ae21c6b8c9ec09a28e0727feea090607b5b5c23bb9590e3"}
Sep 17 23:19:58.144959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.144845    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"
Sep 17 23:19:58.144959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.144938    2580 scope.go:117] "RemoveContainer" containerID="1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e"
Sep 17 23:19:58.154718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.154695    2580 scope.go:117] "RemoveContainer" containerID="1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e"
Sep 17 23:19:58.155038 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:19:58.155010    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e\": container with ID starting with 1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e not found: ID does not exist" containerID="1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e"
Sep 17 23:19:58.155087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.155048    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e"} err="failed to get container status \"1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e\": rpc error: code = NotFound desc = could not find container \"1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e\": container with ID starting with 1367b3936d6b75c6789cbb079a59edd67b88b0219619cf500b961c50f151369e not found: ID does not exist"
Sep 17 23:19:58.163846 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.163821    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"]
Sep 17 23:19:58.168199 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.168176    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6114ed-b5574bd74-82cj8"]
Sep 17 23:19:58.438537 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:58.438502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:59.446723 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:19:59.446680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:00.109825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:00.109764    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4437ed2-2c09-4b1f-b712-05c2f824f24f" path="/var/lib/kubelet/pods/c4437ed2-2c09-4b1f-b712-05c2f824f24f/volumes"
Sep 17 23:20:00.158466 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:00.158413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:00.845975 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:00.845943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:01.737264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:01.737229    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:02.378777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:02.378744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:02.925552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:02.925517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:07.048019 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:07.047988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:07.120300 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:07.120264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:08.853391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:08.853356    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:09.109168 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:09.109080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:09.574329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:09.574299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:09.886335 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:09.886240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:09.977580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:09.977549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:12.052209 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:12.052180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:12.857740 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:12.857689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:13.036977 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:13.036944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:13.134266 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:13.134184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:14.050876 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:14.050844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:14.840250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:14.840211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:15.551323 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:15.551287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:16.235520 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:16.235489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:16.879282 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:16.879243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:17.553646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:17.553612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:17.653979 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:17.653921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:17.999690 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:17.999658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:18.096284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:18.096254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:18.476062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:18.476029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:19.287743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:19.287710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:20.047689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:20.047656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:20.697845 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:20.697785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:23.775343 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:23.775294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:23.857347 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:23.857309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:23.923058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:23.923012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:24.022918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:24.022885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:24.114599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:24.114517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:24.325423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:24.325392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:24.793079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:24.793046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:24.892405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:24.892366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:24.997659 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:24.997621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:25.170455 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:25.170420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:26.038267 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:26.038234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:26.360668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:26.360579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:26.720184 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:26.720150    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:27.382901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:27.382868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:28.147014 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:28.146976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:29.357670 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:29.357637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:30.144557 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:30.144524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:30.923059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:30.923011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:31.589277 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:31.589237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:32.373075 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:32.373040    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:32.473392 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:32.473350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:32.574818 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:32.574768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:32.677772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:32.677730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:32.819990 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:32.819956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:32.903279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:32.903247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:33.153304 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:33.153271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:33.255830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:33.255782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:49.936628 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:49.936592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:51.937246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937206    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"]
Sep 17 23:20:51.937674 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937656    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235c3d05-f364-4981-81cd-7514ad0e1d76" containerName="payload-pre-processing"
Sep 17 23:20:51.937718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937680    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="235c3d05-f364-4981-81cd-7514ad0e1d76" containerName="payload-pre-processing"
Sep 17 23:20:51.937718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937704    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4437ed2-2c09-4b1f-b712-05c2f824f24f" containerName="payload-pre-processing"
Sep 17 23:20:51.937718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937713    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4437ed2-2c09-4b1f-b712-05c2f824f24f" containerName="payload-pre-processing"
Sep 17 23:20:51.937861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca6d3b0-7e6a-4be1-9518-5b42be61c575" containerName="revoke-keys"
Sep 17 23:20:51.937861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937754    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca6d3b0-7e6a-4be1-9518-5b42be61c575" containerName="revoke-keys"
Sep 17 23:20:51.937924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937883    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="235c3d05-f364-4981-81cd-7514ad0e1d76" containerName="payload-pre-processing"
Sep 17 23:20:51.937924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937897    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca6d3b0-7e6a-4be1-9518-5b42be61c575" containerName="revoke-keys"
Sep 17 23:20:51.937924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.937905    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4437ed2-2c09-4b1f-b712-05c2f824f24f" containerName="payload-pre-processing"
Sep 17 23:20:51.943003 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.942984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:51.945369 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.945349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0b4d93\"/\"premium-w0-0b4d93-kserve-self-signed-certs\""
Sep 17 23:20:51.950414 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.950389    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"]
Sep 17 23:20:51.976435 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:51.976411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:52.073571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.073539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0133aa-a194-47ca-8a9a-99c570817943-tls-certs\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.073863 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.073621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj768\" (UniqueName: \"kubernetes.io/projected/6e0133aa-a194-47ca-8a9a-99c570817943-kube-api-access-fj768\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.073863 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.073656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-dshm\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.073863 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.073677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-home\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.073863 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.073707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-tmp-dir\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.073863 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.073740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-model-cache\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.174867 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.174831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0133aa-a194-47ca-8a9a-99c570817943-tls-certs\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.174918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj768\" (UniqueName: \"kubernetes.io/projected/6e0133aa-a194-47ca-8a9a-99c570817943-kube-api-access-fj768\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.174953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-dshm\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.174975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-home\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.175006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-tmp-dir\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175195 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.175049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-model-cache\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175448 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.175414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-home\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175448 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.175432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-tmp-dir\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.175603 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.175455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-model-cache\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.177314 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.177294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-dshm\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.177732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.177714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0133aa-a194-47ca-8a9a-99c570817943-tls-certs\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.183547 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.183524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj768\" (UniqueName: \"kubernetes.io/projected/6e0133aa-a194-47ca-8a9a-99c570817943-kube-api-access-fj768\") pod \"premium-w0-0b4d93-kserve-b5657bf54-27w6d\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") " pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.253737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.253655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:20:52.529819 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:52.529770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"]
Sep 17 23:20:53.442131 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:53.442090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d" event={"ID":"6e0133aa-a194-47ca-8a9a-99c570817943","Type":"ContainerStarted","Data":"6c66f00f798ea96f7a4958aeb16eb27f6d75bffcde4c447ec4efcf9c08f6a83d"}
Sep 17 23:20:53.442131 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:53.442138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d" event={"ID":"6e0133aa-a194-47ca-8a9a-99c570817943","Type":"ContainerStarted","Data":"8fc87c3bd924dbf7dca316c773eb139f2158fb969f781cbfd88d77161fd850c2"}
Sep 17 23:20:53.463072 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:53.463015    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d" podStartSLOduration=2.462999616 podStartE2EDuration="2.462999616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:53.461467938 +0000 UTC m=+863.891327776" watchObservedRunningTime="2026-09-17 23:20:53.462999616 +0000 UTC m=+863.892859463"
Sep 17 23:20:59.083943 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:20:59.083897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:02.252484 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:02.252442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:02.253879 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:02.253853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:21:02.254021 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:02.253895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:21:02.266595 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:02.266573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:21:02.504076 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:02.503996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:21:04.714517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:04.714479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:04.814957 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:04.814925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:04.915616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:04.915565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:15.065675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:15.065637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:20.708840 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:20.708782    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:26.997552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:26.997471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:27.098046 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:27.098013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:27.197681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:27.197645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:30.062781 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:30.062751    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:21:30.063559 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:30.063541    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:21:30.096426 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:30.096393    2580 scope.go:117] "RemoveContainer" containerID="8b0f327d3d22a96921a769280a48c53e881471c064fd29fc7943d91d4749c7e2"
Sep 17 23:21:31.174642 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:31.174612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:37.664443 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:37.664375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:38.482843 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:38.482784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:44.200852 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:44.200786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:44.304248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:44.304192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:44.403370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:44.403335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:44.461880 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:44.461763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:45.139320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:45.139284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:45.231221 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:45.231187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:45.331997 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:45.331963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:46.757853 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:46.757792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:51.522091 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:51.522052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:52.204423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:52.204391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:52.869839 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:52.869779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:53.021947 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:53.021911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:53.523351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:53.523319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:53.744950 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:53.744912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:54.449290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:54.449250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:55.261344 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:55.261314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:59.217718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:59.217675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:59.318616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:59.318580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:59.444155 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:59.444121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:59.734065 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:59.734036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:59.974028 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:21:59.974000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:01.424078 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:01.424035    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:01.515011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:01.514977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:01.591608 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:01.591576    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:01.675000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:01.674912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:01.745642 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:01.745611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:02.015997 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:02.015906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:02.145753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:02.145715    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:02.851959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:02.851926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:03.670730 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:03.670699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:04.377894 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:04.377861    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:05.406779 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:05.406724    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:06.582493 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:06.582463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:07.093889 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:07.093858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:07.194653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:07.194618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:08.042041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:08.042010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:08.947430 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:08.947395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:09.818389 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:09.818358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:10.660647 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:10.660607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:10.758219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:10.758181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:10.858052 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:10.858016    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:11.248566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.248533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:11.335355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.335319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:11.433395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.433341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:11.489387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.489342    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"]
Sep 17 23:22:11.493058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.493033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:11.495287 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.495261    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-700887b8-dockercfg-j9r2n\""
Sep 17 23:22:11.495428 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.495263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-700887b8\""
Sep 17 23:22:11.503438 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.503413    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"]
Sep 17 23:22:11.579368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.579314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897f4532-c035-46f9-ac18-48b8282e4de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:11.579558 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.579456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlkbd\" (UniqueName: \"kubernetes.io/projected/897f4532-c035-46f9-ac18-48b8282e4de6-kube-api-access-nlkbd\") pod \"payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:11.680514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.680466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlkbd\" (UniqueName: \"kubernetes.io/projected/897f4532-c035-46f9-ac18-48b8282e4de6-kube-api-access-nlkbd\") pod \"payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:11.680727 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.680558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897f4532-c035-46f9-ac18-48b8282e4de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:11.681294 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.681272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897f4532-c035-46f9-ac18-48b8282e4de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:11.696623 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.696598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlkbd\" (UniqueName: \"kubernetes.io/projected/897f4532-c035-46f9-ac18-48b8282e4de6-kube-api-access-nlkbd\") pod \"payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:11.804311 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:11.804228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:12.077901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:12.077874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"]
Sep 17 23:22:12.083457 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:22:12.083421    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897f4532_c035_46f9_ac18_48b8282e4de6.slice/crio-0e085a5c53cee2b9c62cfa5cb7239d09b27fdada2729884ccc6fab6b9b740591 WatchSource:0}: Error finding container 0e085a5c53cee2b9c62cfa5cb7239d09b27fdada2729884ccc6fab6b9b740591: Status 404 returned error can't find the container with id 0e085a5c53cee2b9c62cfa5cb7239d09b27fdada2729884ccc6fab6b9b740591
Sep 17 23:22:12.879927 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:12.879890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd" event={"ID":"897f4532-c035-46f9-ac18-48b8282e4de6","Type":"ContainerStarted","Data":"855ef8a8b4027c0041bd199221ff1b5f21537617ef0374f2a736c172af4acb69"}
Sep 17 23:22:12.880140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:12.879947    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:12.880140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:12.879965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd" event={"ID":"897f4532-c035-46f9-ac18-48b8282e4de6","Type":"ContainerStarted","Data":"0e085a5c53cee2b9c62cfa5cb7239d09b27fdada2729884ccc6fab6b9b740591"}
Sep 17 23:22:12.898689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:12.898634    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd" podStartSLOduration=1.8986154229999999 podStartE2EDuration="1.898615423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:12.896614688 +0000 UTC m=+943.326474507" watchObservedRunningTime="2026-09-17 23:22:12.898615423 +0000 UTC m=+943.328475286"
Sep 17 23:22:15.539939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:15.539907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:23.887979 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:23.887948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:27.549789 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:27.549752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:27.771943 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:27.771906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:28.196048 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:28.196012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:28.491690 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:28.491601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:28.916400 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:28.916364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:30.153826 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:30.153736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:30.247616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:30.247561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:30.349208 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:30.349175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:33.247559 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:33.247523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:33.668167 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:33.668126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:34.077503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:34.077415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:34.764646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:34.764612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:35.546379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:35.546348    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:36.081769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:36.081737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:36.218480 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:36.218445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:36.779007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:36.778968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:36.866230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:36.866191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:37.151924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:37.151890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:37.250015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:37.249983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:43.046822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:43.046762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:43.146637 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:43.146600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:44.157502 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:44.157465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:44.853285 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:44.853256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:45.967965 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:45.967932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:46.864325 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:46.864294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:47.659044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:47.658956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:48.584849 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:48.584790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:53.906029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:53.905937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:53.938383 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:53.938349    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"]
Sep 17 23:22:53.938794 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:53.938757    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd" podUID="897f4532-c035-46f9-ac18-48b8282e4de6" containerName="payload-pre-processing" containerID="cri-o://855ef8a8b4027c0041bd199221ff1b5f21537617ef0374f2a736c172af4acb69" gracePeriod=30
Sep 17 23:22:54.006245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.006161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:54.091915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.091884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:54.119129 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.119094    2580 generic.go:356] "Generic (PLEG): container finished" podID="897f4532-c035-46f9-ac18-48b8282e4de6" containerID="855ef8a8b4027c0041bd199221ff1b5f21537617ef0374f2a736c172af4acb69" exitCode=0
Sep 17 23:22:54.119283 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.119163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd" event={"ID":"897f4532-c035-46f9-ac18-48b8282e4de6","Type":"ContainerDied","Data":"855ef8a8b4027c0041bd199221ff1b5f21537617ef0374f2a736c172af4acb69"}
Sep 17 23:22:54.153624 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.153574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:54.217456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.217428    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:54.311669 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.311637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897f4532-c035-46f9-ac18-48b8282e4de6-plugins-config-volume\") pod \"897f4532-c035-46f9-ac18-48b8282e4de6\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") "
Sep 17 23:22:54.311901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.311724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlkbd\" (UniqueName: \"kubernetes.io/projected/897f4532-c035-46f9-ac18-48b8282e4de6-kube-api-access-nlkbd\") pod \"897f4532-c035-46f9-ac18-48b8282e4de6\" (UID: \"897f4532-c035-46f9-ac18-48b8282e4de6\") "
Sep 17 23:22:54.312154 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.312128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897f4532-c035-46f9-ac18-48b8282e4de6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "897f4532-c035-46f9-ac18-48b8282e4de6" (UID: "897f4532-c035-46f9-ac18-48b8282e4de6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:22:54.314019 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.313996    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897f4532-c035-46f9-ac18-48b8282e4de6-kube-api-access-nlkbd" (OuterVolumeSpecName: "kube-api-access-nlkbd") pod "897f4532-c035-46f9-ac18-48b8282e4de6" (UID: "897f4532-c035-46f9-ac18-48b8282e4de6"). InnerVolumeSpecName "kube-api-access-nlkbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:22:54.412578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.412540    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897f4532-c035-46f9-ac18-48b8282e4de6-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:22:54.412578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.412573    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlkbd\" (UniqueName: \"kubernetes.io/projected/897f4532-c035-46f9-ac18-48b8282e4de6-kube-api-access-nlkbd\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:22:54.454817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.454722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:54.556269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:54.556233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:55.125599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:55.125566    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"
Sep 17 23:22:55.125599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:55.125593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd" event={"ID":"897f4532-c035-46f9-ac18-48b8282e4de6","Type":"ContainerDied","Data":"0e085a5c53cee2b9c62cfa5cb7239d09b27fdada2729884ccc6fab6b9b740591"}
Sep 17 23:22:55.126229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:55.125662    2580 scope.go:117] "RemoveContainer" containerID="855ef8a8b4027c0041bd199221ff1b5f21537617ef0374f2a736c172af4acb69"
Sep 17 23:22:55.146486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:55.146448    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"]
Sep 17 23:22:55.148928 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:55.148887    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-700887b8-76d75bfd86-drlcd"]
Sep 17 23:22:55.262677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:55.262649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:55.930725 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:55.930695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:56.110054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:56.110022    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897f4532-c035-46f9-ac18-48b8282e4de6" path="/var/lib/kubelet/pods/897f4532-c035-46f9-ac18-48b8282e4de6/volumes"
Sep 17 23:22:56.747895 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:56.747860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:57.394425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:22:57.394384    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:00.201881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:00.201848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:00.272237 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:00.272194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:00.467498 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:00.467392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:00.850646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:00.850517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:00.946988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:00.946953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:01.046055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:01.045985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:04.066279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:04.066233    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:06.147633 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:06.147569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:07.293763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:07.293719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:07.395860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:07.395783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:07.499019 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:07.498985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:07.600095 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:07.600003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:07.696746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:07.696711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:09.342222 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:09.342128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:09.443795 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:09.443760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:10.413986 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:10.413942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:11.070617 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:11.070577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:11.830965 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:11.830932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:13.370952 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:13.370892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:13.435373 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:13.435344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:14.110795 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:14.110758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:14.273087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:14.273045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:14.593712 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:14.593677    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:14.952854 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:14.952824    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:17.285772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:17.285726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:17.983822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:17.983776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:18.084040 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:18.084003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:19.036955 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:19.036918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:19.811512 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:19.811479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:20.642331 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:20.642292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.315942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:21.315899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.413954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:21.413915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.473512 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:21.473477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.772508 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:21.772474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:21.865034 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:21.865004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:31.291285 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:31.291232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:38.257473 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:38.257441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:38.522527 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:38.522436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:38.638762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:38.638728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:42.162465 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:42.162437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:43.541282 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:43.541253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:43.683179 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:43.683136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:44.352085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:44.352049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:45.128822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:45.128772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:45.779036 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:45.778990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:45.879570 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:45.879539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:46.635410 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:46.635359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:47.346100 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:47.346064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:48.026618 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:48.026587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:48.339356 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:48.339245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:48.663916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:48.663876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:49.420186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:49.420127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:50.038556 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:50.038527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:52.037334 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:52.037302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:52.162622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:52.162584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:52.250781 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:52.250747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:52.876375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:52.876342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:53.841120 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:53.841083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:54.522496 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:54.522465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:55.160549 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:55.160516    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:55.960967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:55.960932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:56.059688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.059657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:56.269765 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.269676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:56.542067 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.541980    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:56.670404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.670371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:56.843757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.843679    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"]
Sep 17 23:23:56.844172 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.844157    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897f4532-c035-46f9-ac18-48b8282e4de6" containerName="payload-pre-processing"
Sep 17 23:23:56.844224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.844176    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="897f4532-c035-46f9-ac18-48b8282e4de6" containerName="payload-pre-processing"
Sep 17 23:23:56.844308 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.844295    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="897f4532-c035-46f9-ac18-48b8282e4de6" containerName="payload-pre-processing"
Sep 17 23:23:56.847874 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.847852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:56.850388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.850362    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-55130c\""
Sep 17 23:23:56.850539 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.850402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-55130c-dockercfg-fq566\""
Sep 17 23:23:56.858101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.858077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"]
Sep 17 23:23:56.947322 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.947281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfbface-810f-497c-a320-63774cf51403-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:56.947515 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:56.947408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httwc\" (UniqueName: \"kubernetes.io/projected/8bfbface-810f-497c-a320-63774cf51403-kube-api-access-httwc\") pod \"payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:57.048998 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:57.048951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfbface-810f-497c-a320-63774cf51403-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:57.049180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:57.049081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httwc\" (UniqueName: \"kubernetes.io/projected/8bfbface-810f-497c-a320-63774cf51403-kube-api-access-httwc\") pod \"payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:57.050190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:57.050149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfbface-810f-497c-a320-63774cf51403-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:57.057931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:57.057900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-httwc\" (UniqueName: \"kubernetes.io/projected/8bfbface-810f-497c-a320-63774cf51403-kube-api-access-httwc\") pod \"payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:57.160246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:57.160207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:57.434333 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:57.434306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"]
Sep 17 23:23:57.436487 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:23:57.436457    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfbface_810f_497c_a320_63774cf51403.slice/crio-02f043ad5159bce788d866814d8fecceadc2a874b41fd8f240a2b590270c29ab WatchSource:0}: Error finding container 02f043ad5159bce788d866814d8fecceadc2a874b41fd8f240a2b590270c29ab: Status 404 returned error can't find the container with id 02f043ad5159bce788d866814d8fecceadc2a874b41fd8f240a2b590270c29ab
Sep 17 23:23:57.465033 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:57.465006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7" event={"ID":"8bfbface-810f-497c-a320-63774cf51403","Type":"ContainerStarted","Data":"02f043ad5159bce788d866814d8fecceadc2a874b41fd8f240a2b590270c29ab"}
Sep 17 23:23:58.471414 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:58.471376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7" event={"ID":"8bfbface-810f-497c-a320-63774cf51403","Type":"ContainerStarted","Data":"da134918811af9b92a8848ae6a5ce2287596ccc5f9649c24eca03f9822f80423"}
Sep 17 23:23:58.471783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:58.471464    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:23:58.488481 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:23:58.488434    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7" podStartSLOduration=2.488418211 podStartE2EDuration="2.488418211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:58.48712483 +0000 UTC m=+1048.916984649" watchObservedRunningTime="2026-09-17 23:23:58.488418211 +0000 UTC m=+1048.918278027"
Sep 17 23:24:09.479673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:09.479641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:24:17.161412 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:17.161372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"]
Sep 17 23:24:17.180866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:17.180821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:18.208626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:18.208594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:19.090706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:19.090676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:19.800695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:19.800662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:20.647145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:20.647108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:21.013425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:21.013337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:21.716756 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:21.716680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:22.221611 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:22.221581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:22.464026 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:22.463995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:23.181923 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:23.181889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:23.955895 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:23.955860    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:24.640100 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:24.640069    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:25.363646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:25.363604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:29.771334 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:29.771302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:30.137414 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.137390    2580 scope.go:117] "RemoveContainer" containerID="3596fc5259ef5f4d96bd5d147e45696596a65146ef69fe6560a052223b35605a"
Sep 17 23:24:30.416251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.416172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:30.444937 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.444895    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"]
Sep 17 23:24:30.445218 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.445186    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5" podUID="a1a93a76-0c68-4a48-a983-ea597a78ec2c" containerName="payload-processing" containerID="cri-o://8051a969136ca38006a14c5535a010f4f650970d4ad504d566bb3bc5073f1564" gracePeriod=30
Sep 17 23:24:30.514173 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.514134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:30.614830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.614769    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:30.651422 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.651392    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1a93a76-0c68-4a48-a983-ea597a78ec2c" containerID="8051a969136ca38006a14c5535a010f4f650970d4ad504d566bb3bc5073f1564" exitCode=0
Sep 17 23:24:30.651569 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.651457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5" event={"ID":"a1a93a76-0c68-4a48-a983-ea597a78ec2c","Type":"ContainerDied","Data":"8051a969136ca38006a14c5535a010f4f650970d4ad504d566bb3bc5073f1564"}
Sep 17 23:24:30.713967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.713926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:30.722341 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.722306    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:24:30.792321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.792288    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a93a76-0c68-4a48-a983-ea597a78ec2c-plugins-config-volume\") pod \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") "
Sep 17 23:24:30.792682 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.792376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5brf\" (UniqueName: \"kubernetes.io/projected/a1a93a76-0c68-4a48-a983-ea597a78ec2c-kube-api-access-w5brf\") pod \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\" (UID: \"a1a93a76-0c68-4a48-a983-ea597a78ec2c\") "
Sep 17 23:24:30.792724 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.792702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a93a76-0c68-4a48-a983-ea597a78ec2c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1a93a76-0c68-4a48-a983-ea597a78ec2c" (UID: "a1a93a76-0c68-4a48-a983-ea597a78ec2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:30.794660 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.794638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a93a76-0c68-4a48-a983-ea597a78ec2c-kube-api-access-w5brf" (OuterVolumeSpecName: "kube-api-access-w5brf") pod "a1a93a76-0c68-4a48-a983-ea597a78ec2c" (UID: "a1a93a76-0c68-4a48-a983-ea597a78ec2c"). InnerVolumeSpecName "kube-api-access-w5brf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:30.893251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.893211    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5brf\" (UniqueName: \"kubernetes.io/projected/a1a93a76-0c68-4a48-a983-ea597a78ec2c-kube-api-access-w5brf\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:30.893251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.893242    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a93a76-0c68-4a48-a983-ea597a78ec2c-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:30.976999 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:30.976916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:31.077965 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.077932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:31.658755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.658727    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"
Sep 17 23:24:31.658950 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.658729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5" event={"ID":"a1a93a76-0c68-4a48-a983-ea597a78ec2c","Type":"ContainerDied","Data":"95869a83b0d81d7b05412c597cf71378a5c3a89b863b344ddff51a2feec1deb4"}
Sep 17 23:24:31.658950 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.658866    2580 scope.go:117] "RemoveContainer" containerID="8051a969136ca38006a14c5535a010f4f650970d4ad504d566bb3bc5073f1564"
Sep 17 23:24:31.682599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.682577    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"]
Sep 17 23:24:31.686746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.686723    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7aa654-5995d54bbf-kw9n5"]
Sep 17 23:24:31.693564 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.693540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:31.781762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:31.781726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:32.009085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.009002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:32.109402 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.109361    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a93a76-0c68-4a48-a983-ea597a78ec2c" path="/var/lib/kubelet/pods/a1a93a76-0c68-4a48-a983-ea597a78ec2c/volumes"
Sep 17 23:24:32.182061 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.182014    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff" podUID="2a3c8049-f61e-499b-8bc5-550a56268f12" containerName="main" containerID="cri-o://d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650" gracePeriod=45
Sep 17 23:24:32.436433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.436411    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:24:32.507112 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507079    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f279n\" (UniqueName: \"kubernetes.io/projected/2a3c8049-f61e-499b-8bc5-550a56268f12-kube-api-access-f279n\") pod \"2a3c8049-f61e-499b-8bc5-550a56268f12\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") "
Sep 17 23:24:32.507276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3c8049-f61e-499b-8bc5-550a56268f12-tls-certs\") pod \"2a3c8049-f61e-499b-8bc5-550a56268f12\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") "
Sep 17 23:24:32.507276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-tmp-dir\") pod \"2a3c8049-f61e-499b-8bc5-550a56268f12\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") "
Sep 17 23:24:32.507276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507185    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-dshm\") pod \"2a3c8049-f61e-499b-8bc5-550a56268f12\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") "
Sep 17 23:24:32.507276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507237    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-home\") pod \"2a3c8049-f61e-499b-8bc5-550a56268f12\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") "
Sep 17 23:24:32.507517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-model-cache\") pod \"2a3c8049-f61e-499b-8bc5-550a56268f12\" (UID: \"2a3c8049-f61e-499b-8bc5-550a56268f12\") "
Sep 17 23:24:32.507517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a3c8049-f61e-499b-8bc5-550a56268f12" (UID: "2a3c8049-f61e-499b-8bc5-550a56268f12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.507621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507544    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-home" (OuterVolumeSpecName: "home") pod "2a3c8049-f61e-499b-8bc5-550a56268f12" (UID: "2a3c8049-f61e-499b-8bc5-550a56268f12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.507674 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a3c8049-f61e-499b-8bc5-550a56268f12" (UID: "2a3c8049-f61e-499b-8bc5-550a56268f12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.507748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507733    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.507833 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507754    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.507833 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.507766    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.509455 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.509436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3c8049-f61e-499b-8bc5-550a56268f12-kube-api-access-f279n" (OuterVolumeSpecName: "kube-api-access-f279n") pod "2a3c8049-f61e-499b-8bc5-550a56268f12" (UID: "2a3c8049-f61e-499b-8bc5-550a56268f12"). InnerVolumeSpecName "kube-api-access-f279n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:32.509853 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.509835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3c8049-f61e-499b-8bc5-550a56268f12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a3c8049-f61e-499b-8bc5-550a56268f12" (UID: "2a3c8049-f61e-499b-8bc5-550a56268f12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:32.509922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.509854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-dshm" (OuterVolumeSpecName: "dshm") pod "2a3c8049-f61e-499b-8bc5-550a56268f12" (UID: "2a3c8049-f61e-499b-8bc5-550a56268f12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.533501 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.533470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:32.609301 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.609197    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f279n\" (UniqueName: \"kubernetes.io/projected/2a3c8049-f61e-499b-8bc5-550a56268f12-kube-api-access-f279n\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.609301 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.609236    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3c8049-f61e-499b-8bc5-550a56268f12-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.609301 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.609260    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3c8049-f61e-499b-8bc5-550a56268f12-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.633421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.633389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:32.666733 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.666703    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a3c8049-f61e-499b-8bc5-550a56268f12" containerID="d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650" exitCode=0
Sep 17 23:24:32.666931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.666760    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"
Sep 17 23:24:32.666931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.666781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff" event={"ID":"2a3c8049-f61e-499b-8bc5-550a56268f12","Type":"ContainerDied","Data":"d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650"}
Sep 17 23:24:32.666931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.666829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff" event={"ID":"2a3c8049-f61e-499b-8bc5-550a56268f12","Type":"ContainerDied","Data":"d92a78419f4cf8f5fa96049734579130537bdc7815938a6997dac772a43b08f3"}
Sep 17 23:24:32.666931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.666845    2580 scope.go:117] "RemoveContainer" containerID="d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650"
Sep 17 23:24:32.676373 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.676357    2580 scope.go:117] "RemoveContainer" containerID="d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650"
Sep 17 23:24:32.676639 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:24:32.676620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650\": container with ID starting with d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650 not found: ID does not exist" containerID="d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650"
Sep 17 23:24:32.676686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.676648    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650"} err="failed to get container status \"d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650\": rpc error: code = NotFound desc = could not find container \"d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650\": container with ID starting with d38c209c3793e0b7f935a5d7daba9d5694e7a63b469565d780f09b8f4dc9b650 not found: ID does not exist"
Sep 17 23:24:32.688267 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.688235    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"]
Sep 17 23:24:32.691445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:32.691424    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7aa654/test-model-7aa654-kserve-55cccdd465-2vkff"]
Sep 17 23:24:33.358290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:33.358256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:34.012694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:34.012660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:34.109093 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:34.109059    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3c8049-f61e-499b-8bc5-550a56268f12" path="/var/lib/kubelet/pods/2a3c8049-f61e-499b-8bc5-550a56268f12/volumes"
Sep 17 23:24:34.781668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:34.781635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:35.041571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:35.041484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:35.141620 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:35.141584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:35.773657 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:35.773622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:36.634773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:36.634740    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:37.117848 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:37.117786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:37.216563 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:37.216530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:37.371392 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:37.371300    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:37.469861 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:37.469830    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:38.256069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:38.256036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:38.389885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:38.389846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:39.430684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:39.430643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.148888 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.148838    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.248054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.248013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.348732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.348687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.657774 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.657734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.754019 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.753980    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.788525 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.788493    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"]
Sep 17 23:24:40.788916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.788900    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a93a76-0c68-4a48-a983-ea597a78ec2c" containerName="payload-processing"
Sep 17 23:24:40.788916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.788914    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a93a76-0c68-4a48-a983-ea597a78ec2c" containerName="payload-processing"
Sep 17 23:24:40.789101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.788927    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3c8049-f61e-499b-8bc5-550a56268f12" containerName="main"
Sep 17 23:24:40.789101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.788933    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3c8049-f61e-499b-8bc5-550a56268f12" containerName="main"
Sep 17 23:24:40.789101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.789004    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3c8049-f61e-499b-8bc5-550a56268f12" containerName="main"
Sep 17 23:24:40.789101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.789012    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a93a76-0c68-4a48-a983-ea597a78ec2c" containerName="payload-processing"
Sep 17 23:24:40.793423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.793398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:40.796572 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.796548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6824f488\""
Sep 17 23:24:40.796668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.796617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6824f488-dockercfg-7rwsj\""
Sep 17 23:24:40.805842 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.805791    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"]
Sep 17 23:24:40.887998 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.887954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf6c118-b82e-4b78-823e-d9395e34ed40-plugins-config-volume\") pod \"payload-processing-e2e-mt-6824f488-847dd44849-cpgg7\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") " pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:40.888216 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.888072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpx64\" (UniqueName: \"kubernetes.io/projected/4cf6c118-b82e-4b78-823e-d9395e34ed40-kube-api-access-dpx64\") pod \"payload-processing-e2e-mt-6824f488-847dd44849-cpgg7\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") " pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:40.988966 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.988869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpx64\" (UniqueName: \"kubernetes.io/projected/4cf6c118-b82e-4b78-823e-d9395e34ed40-kube-api-access-dpx64\") pod \"payload-processing-e2e-mt-6824f488-847dd44849-cpgg7\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") " pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:40.988966 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.988960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf6c118-b82e-4b78-823e-d9395e34ed40-plugins-config-volume\") pod \"payload-processing-e2e-mt-6824f488-847dd44849-cpgg7\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") " pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:40.989612 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.989595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf6c118-b82e-4b78-823e-d9395e34ed40-plugins-config-volume\") pod \"payload-processing-e2e-mt-6824f488-847dd44849-cpgg7\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") " pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:40.997831 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:40.997772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpx64\" (UniqueName: \"kubernetes.io/projected/4cf6c118-b82e-4b78-823e-d9395e34ed40-kube-api-access-dpx64\") pod \"payload-processing-e2e-mt-6824f488-847dd44849-cpgg7\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") " pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:41.103943 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:41.103915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:41.134939 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:41.134905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:41.380329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:41.380297    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"]
Sep 17 23:24:41.382994 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:24:41.382966    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf6c118_b82e_4b78_823e_d9395e34ed40.slice/crio-388d632fd67d67697e7801034229a1a016123a0c54e789509f2af4bfbe3753f1 WatchSource:0}: Error finding container 388d632fd67d67697e7801034229a1a016123a0c54e789509f2af4bfbe3753f1: Status 404 returned error can't find the container with id 388d632fd67d67697e7801034229a1a016123a0c54e789509f2af4bfbe3753f1
Sep 17 23:24:41.718562 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:41.718525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7" event={"ID":"4cf6c118-b82e-4b78-823e-d9395e34ed40","Type":"ContainerStarted","Data":"a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd"}
Sep 17 23:24:41.718959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:41.718585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7" event={"ID":"4cf6c118-b82e-4b78-823e-d9395e34ed40","Type":"ContainerStarted","Data":"388d632fd67d67697e7801034229a1a016123a0c54e789509f2af4bfbe3753f1"}
Sep 17 23:24:41.718959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:41.718615    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:41.735217 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:41.735158    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7" podStartSLOduration=1.735139433 podStartE2EDuration="1.735139433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:41.734829875 +0000 UTC m=+1092.164689687" watchObservedRunningTime="2026-09-17 23:24:41.735139433 +0000 UTC m=+1092.164999252"
Sep 17 23:24:42.136995 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:42.136894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:42.856112 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:42.856082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:43.521188 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:43.521161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:44.257764 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:44.257730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:47.843038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:47.842995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:48.406317 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.406286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:48.434870 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.434838    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"]
Sep 17 23:24:48.435122 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.435094    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld" podUID="65eeba0b-33ed-4ae2-a820-dac3b240fff7" containerName="payload-processing" containerID="cri-o://1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8" gracePeriod=30
Sep 17 23:24:48.441166 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.441137    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"]
Sep 17 23:24:48.441390 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.441358    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v" podUID="001690c8-6863-414e-9c30-d41dbcfcf11b" containerName="payload-pre-processing" containerID="cri-o://ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420" gracePeriod=30
Sep 17 23:24:48.479726 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.479673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:48.541922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.541873    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:48.606177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.606094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:48.663364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.663261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:48.726346 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.726315    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:24:48.728348 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.728331    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:24:48.757296 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.757268    2580 generic.go:356] "Generic (PLEG): container finished" podID="65eeba0b-33ed-4ae2-a820-dac3b240fff7" containerID="1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8" exitCode=0
Sep 17 23:24:48.757404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.757310    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"
Sep 17 23:24:48.757404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.757354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld" event={"ID":"65eeba0b-33ed-4ae2-a820-dac3b240fff7","Type":"ContainerDied","Data":"1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8"}
Sep 17 23:24:48.757491 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.757404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld" event={"ID":"65eeba0b-33ed-4ae2-a820-dac3b240fff7","Type":"ContainerDied","Data":"dc5aaee93394fb611f2e04865f9a4b8970b3b27030b530fc0114f9c7f5655543"}
Sep 17 23:24:48.757491 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.757432    2580 scope.go:117] "RemoveContainer" containerID="1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8"
Sep 17 23:24:48.758640 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.758619    2580 generic.go:356] "Generic (PLEG): container finished" podID="001690c8-6863-414e-9c30-d41dbcfcf11b" containerID="ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420" exitCode=0
Sep 17 23:24:48.758724 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.758653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v" event={"ID":"001690c8-6863-414e-9c30-d41dbcfcf11b","Type":"ContainerDied","Data":"ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420"}
Sep 17 23:24:48.758724 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.758666    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"
Sep 17 23:24:48.758724 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.758703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v" event={"ID":"001690c8-6863-414e-9c30-d41dbcfcf11b","Type":"ContainerDied","Data":"fa3ad7f8caba0ff1ebca5c2efd8fee35bdc17eedadf3c62d7fa3dc89d33e1d19"}
Sep 17 23:24:48.767060 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.767024    2580 scope.go:117] "RemoveContainer" containerID="1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8"
Sep 17 23:24:48.767328 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:24:48.767302    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8\": container with ID starting with 1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8 not found: ID does not exist" containerID="1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8"
Sep 17 23:24:48.767397 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.767333    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8"} err="failed to get container status \"1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8\": rpc error: code = NotFound desc = could not find container \"1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8\": container with ID starting with 1b11dc5f6daf7ae39803feef117b5acb60a23436b9099daa3429910c875b3aa8 not found: ID does not exist"
Sep 17 23:24:48.767397 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.767350    2580 scope.go:117] "RemoveContainer" containerID="ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420"
Sep 17 23:24:48.777079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.777060    2580 scope.go:117] "RemoveContainer" containerID="ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420"
Sep 17 23:24:48.777408 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:24:48.777388    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420\": container with ID starting with ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420 not found: ID does not exist" containerID="ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420"
Sep 17 23:24:48.777516 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.777414    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420"} err="failed to get container status \"ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420\": rpc error: code = NotFound desc = could not find container \"ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420\": container with ID starting with ed16103da6d70c289110b3ef198b55c30ed895a6308f4d2cbcc175c580075420 not found: ID does not exist"
Sep 17 23:24:48.869714 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.869684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49rvg\" (UniqueName: \"kubernetes.io/projected/001690c8-6863-414e-9c30-d41dbcfcf11b-kube-api-access-49rvg\") pod \"001690c8-6863-414e-9c30-d41dbcfcf11b\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") "
Sep 17 23:24:48.870164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.869772    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm2t8\" (UniqueName: \"kubernetes.io/projected/65eeba0b-33ed-4ae2-a820-dac3b240fff7-kube-api-access-bm2t8\") pod \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") "
Sep 17 23:24:48.870164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.869830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eeba0b-33ed-4ae2-a820-dac3b240fff7-plugins-config-volume\") pod \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\" (UID: \"65eeba0b-33ed-4ae2-a820-dac3b240fff7\") "
Sep 17 23:24:48.870164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.869859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001690c8-6863-414e-9c30-d41dbcfcf11b-plugins-config-volume\") pod \"001690c8-6863-414e-9c30-d41dbcfcf11b\" (UID: \"001690c8-6863-414e-9c30-d41dbcfcf11b\") "
Sep 17 23:24:48.870327 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.870221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65eeba0b-33ed-4ae2-a820-dac3b240fff7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65eeba0b-33ed-4ae2-a820-dac3b240fff7" (UID: "65eeba0b-33ed-4ae2-a820-dac3b240fff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:48.870327 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.870258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001690c8-6863-414e-9c30-d41dbcfcf11b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "001690c8-6863-414e-9c30-d41dbcfcf11b" (UID: "001690c8-6863-414e-9c30-d41dbcfcf11b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:48.872096 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.872074    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001690c8-6863-414e-9c30-d41dbcfcf11b-kube-api-access-49rvg" (OuterVolumeSpecName: "kube-api-access-49rvg") pod "001690c8-6863-414e-9c30-d41dbcfcf11b" (UID: "001690c8-6863-414e-9c30-d41dbcfcf11b"). InnerVolumeSpecName "kube-api-access-49rvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:48.872219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.872100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65eeba0b-33ed-4ae2-a820-dac3b240fff7-kube-api-access-bm2t8" (OuterVolumeSpecName: "kube-api-access-bm2t8") pod "65eeba0b-33ed-4ae2-a820-dac3b240fff7" (UID: "65eeba0b-33ed-4ae2-a820-dac3b240fff7"). InnerVolumeSpecName "kube-api-access-bm2t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:48.971180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.971093    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm2t8\" (UniqueName: \"kubernetes.io/projected/65eeba0b-33ed-4ae2-a820-dac3b240fff7-kube-api-access-bm2t8\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.971180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.971125    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eeba0b-33ed-4ae2-a820-dac3b240fff7-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.971180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.971134    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001690c8-6863-414e-9c30-d41dbcfcf11b-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.971180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.971144    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49rvg\" (UniqueName: \"kubernetes.io/projected/001690c8-6863-414e-9c30-d41dbcfcf11b-kube-api-access-49rvg\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.994994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:48.994964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:49.080964 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:49.080930    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"]
Sep 17 23:24:49.084009 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:49.083982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:49.084891 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:49.084871    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8fc0d3-558b678bd7-svxld"]
Sep 17 23:24:49.095724 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:49.095694    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"]
Sep 17 23:24:49.099777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:49.099750    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8fc0d3-7596c7c56b-h5z8v"]
Sep 17 23:24:49.735698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:49.735611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:50.108835 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:50.108744    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001690c8-6863-414e-9c30-d41dbcfcf11b" path="/var/lib/kubelet/pods/001690c8-6863-414e-9c30-d41dbcfcf11b/volumes"
Sep 17 23:24:50.109185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:50.109161    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65eeba0b-33ed-4ae2-a820-dac3b240fff7" path="/var/lib/kubelet/pods/65eeba0b-33ed-4ae2-a820-dac3b240fff7/volumes"
Sep 17 23:24:50.560242 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:50.560210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:51.358896 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:51.358863    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.211351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:52.211322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.726409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:52.726374    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:24:53.045197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:53.045107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:53.686893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:53.686857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:54.061896 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:54.061781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:54.385838 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:54.385727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:54.845391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:54.845359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:55.050209 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:55.050171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:55.736942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:55.736907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.120466 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:56.120381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.223960 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:56.223925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.401822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:56.401766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.708918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:56.708833    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.944817 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:56.944776    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:57.368409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:57.368371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:57.598003 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:57.597965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:59.555564 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:24:59.555529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:00.252964 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:00.252925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:00.347709 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:00.347673    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:01.184746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:01.184706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:02.052187 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:02.052153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:02.960973 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:02.960942    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:03.744451 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:03.744413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:04.660027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:04.659995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:04.761500 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:04.761469    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:04.966120 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:04.966037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:05.153544 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.153513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:05.254931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.254843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:05.285210 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.285181    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"]
Sep 17 23:25:05.285662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.285645    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65eeba0b-33ed-4ae2-a820-dac3b240fff7" containerName="payload-processing"
Sep 17 23:25:05.285662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.285664    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65eeba0b-33ed-4ae2-a820-dac3b240fff7" containerName="payload-processing"
Sep 17 23:25:05.285836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.285682    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001690c8-6863-414e-9c30-d41dbcfcf11b" containerName="payload-pre-processing"
Sep 17 23:25:05.285836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.285689    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="001690c8-6863-414e-9c30-d41dbcfcf11b" containerName="payload-pre-processing"
Sep 17 23:25:05.285836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.285828    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65eeba0b-33ed-4ae2-a820-dac3b240fff7" containerName="payload-processing"
Sep 17 23:25:05.285973 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.285854    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="001690c8-6863-414e-9c30-d41dbcfcf11b" containerName="payload-pre-processing"
Sep 17 23:25:05.289049 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.289030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.291302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.291278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e5aae7-dockercfg-mnwl7\""
Sep 17 23:25:05.291429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.291306    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e5aae7\""
Sep 17 23:25:05.299582 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.299559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"]
Sep 17 23:25:05.341269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.341231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.341269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.341271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvs2m\" (UniqueName: \"kubernetes.io/projected/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-kube-api-access-bvs2m\") pod \"payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.442447 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.442407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.442602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.442454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvs2m\" (UniqueName: \"kubernetes.io/projected/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-kube-api-access-bvs2m\") pod \"payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.443175 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.443155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.453885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.453857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvs2m\" (UniqueName: \"kubernetes.io/projected/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-kube-api-access-bvs2m\") pod \"payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.602113 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.602010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:05.877903 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:05.877876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"]
Sep 17 23:25:05.879774 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:25:05.879744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323da953_19d0_4aa5_8d2a_d25f1ccc71e9.slice/crio-cc42f48d48c3c14262fc3204b8c98022bf98c5142497a672f10eb96e68780567 WatchSource:0}: Error finding container cc42f48d48c3c14262fc3204b8c98022bf98c5142497a672f10eb96e68780567: Status 404 returned error can't find the container with id cc42f48d48c3c14262fc3204b8c98022bf98c5142497a672f10eb96e68780567
Sep 17 23:25:06.861043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:06.860999    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs" event={"ID":"323da953-19d0-4aa5-8d2a-d25f1ccc71e9","Type":"ContainerStarted","Data":"c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d"}
Sep 17 23:25:06.861226 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:06.861065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs" event={"ID":"323da953-19d0-4aa5-8d2a-d25f1ccc71e9","Type":"ContainerStarted","Data":"cc42f48d48c3c14262fc3204b8c98022bf98c5142497a672f10eb96e68780567"}
Sep 17 23:25:06.861226 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:06.861197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:06.878480 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:06.878426    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs" podStartSLOduration=1.8784102470000001 podStartE2EDuration="1.878410247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:06.877266199 +0000 UTC m=+1117.307126018" watchObservedRunningTime="2026-09-17 23:25:06.878410247 +0000 UTC m=+1117.308270108"
Sep 17 23:25:17.868437 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:17.868404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:25:20.171341 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:20.171305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:25.585408 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:25.585377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:27.239679 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:27.239648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:28.162761 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:28.162725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:28.890366 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:28.890303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:29.681646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:29.681598    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:30.290343 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.290309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:30.450581 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.450496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:30.550958 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.550928    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:30.650834 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.650786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:30.911885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.911849    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"]
Sep 17 23:25:30.917713 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.917689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.919980 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.919960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6275e9-openshift-default-dockercfg-n7ng2\""
Sep 17 23:25:30.923710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.923676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"]
Sep 17 23:25:30.946413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.946376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:30.997101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-certs\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-token\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997299 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b101bd-be25-4f15-b585-5f45a64b1dcd-istiod-ca-cert\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997299 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-envoy\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqtv\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-kube-api-access-glqtv\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997435 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-podinfo\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997435 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-data\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997558 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-credential-socket\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:30.997558 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:30.997475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-socket\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.098938 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.098902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-envoy\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.098947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqtv\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-kube-api-access-glqtv\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.098975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-podinfo\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-data\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-credential-socket\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-socket\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-certs\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-token\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099444 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b101bd-be25-4f15-b585-5f45a64b1dcd-istiod-ca-cert\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099610 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-data\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099610 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-socket\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-certs\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.099720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.099645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-credential-socket\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.100290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.100266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b101bd-be25-4f15-b585-5f45a64b1dcd-istiod-ca-cert\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.101441 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.101418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-envoy\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.101567 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.101551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-podinfo\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.106783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.106758    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqtv\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-kube-api-access-glqtv\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.107171 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.107153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-token\") pod \"e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") " pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.230697 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.230596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:31.437160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.437099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:31.515585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.515558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"]
Sep 17 23:25:31.517667 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:25:31.517636    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b101bd_be25_4f15_b585_5f45a64b1dcd.slice/crio-96984f6467ff269f7fe91e5b742f1707a7dba398278c0005494a38a29b20442a WatchSource:0}: Error finding container 96984f6467ff269f7fe91e5b742f1707a7dba398278c0005494a38a29b20442a: Status 404 returned error can't find the container with id 96984f6467ff269f7fe91e5b742f1707a7dba398278c0005494a38a29b20442a
Sep 17 23:25:31.517885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.517868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:31.520228 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.520188    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:25:31.520328 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.520281    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:25:31.520371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.520323    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:25:31.997008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.996969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" event={"ID":"32b101bd-be25-4f15-b585-5f45a64b1dcd","Type":"ContainerStarted","Data":"de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f"}
Sep 17 23:25:31.997230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:31.997031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" event={"ID":"32b101bd-be25-4f15-b585-5f45a64b1dcd","Type":"ContainerStarted","Data":"96984f6467ff269f7fe91e5b742f1707a7dba398278c0005494a38a29b20442a"}
Sep 17 23:25:32.026650 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:32.026588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podStartSLOduration=2.026567448 podStartE2EDuration="2.026567448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:32.023546665 +0000 UTC m=+1142.453406498" watchObservedRunningTime="2026-09-17 23:25:32.026567448 +0000 UTC m=+1142.456427267"
Sep 17 23:25:32.227719 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:32.227685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:32.231579 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:32.231558    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:32.236990 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:32.236962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:33.002690 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:33.002659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:33.003690 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:33.003670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:25:33.241254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:33.241201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:33.839936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:33.839905    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:33.959498 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:33.959447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:34.108272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:34.108164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:34.838437 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:34.838406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:35.694531 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:35.694496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.571041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:36.570996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:37.274662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:37.274623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:38.095019 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:38.094980    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:38.869662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:38.869629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:38.946245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:38.946212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:39.189555 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:39.189527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:39.349511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:39.349484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:39.456085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:39.455994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:47.523702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:47.523666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:47.580773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:47.580737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:50.433183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.433145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"]
Sep 17 23:25:50.437837 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.437794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.440105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.440082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6824f488\"/\"default-dockercfg-4l4kv\""
Sep 17 23:25:50.440218 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.440082    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6824f488\"/\"openshift-service-ca.crt\""
Sep 17 23:25:50.440924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.440909    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6824f488\"/\"kube-root-ca.crt\""
Sep 17 23:25:50.441049 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.441028    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6824f488\"/\"e2e-lifecycle-model-6824f488-kserve-self-signed-certs\""
Sep 17 23:25:50.445520 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.445490    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"]
Sep 17 23:25:50.450211 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.450184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:50.489498 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.489459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a95d7c-e171-4201-9d62-8dca8054e498-tls-certs\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.489672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.489578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-dshm\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.489672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.489635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5dd\" (UniqueName: \"kubernetes.io/projected/51a95d7c-e171-4201-9d62-8dca8054e498-kube-api-access-4l5dd\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.489786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.489677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-model-cache\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.489786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.489698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-tmp-dir\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.489786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.489759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-home\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591088 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-home\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a95d7c-e171-4201-9d62-8dca8054e498-tls-certs\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-dshm\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5dd\" (UniqueName: \"kubernetes.io/projected/51a95d7c-e171-4201-9d62-8dca8054e498-kube-api-access-4l5dd\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-model-cache\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591286 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-tmp-dir\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591565 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-home\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-tmp-dir\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.591666 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.591620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-model-cache\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.593719 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.593694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-dshm\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.593893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.593873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a95d7c-e171-4201-9d62-8dca8054e498-tls-certs\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.599650 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.599620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5dd\" (UniqueName: \"kubernetes.io/projected/51a95d7c-e171-4201-9d62-8dca8054e498-kube-api-access-4l5dd\") pod \"e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") " pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:50.749258 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:50.749167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:25:51.026042 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:51.026014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"]
Sep 17 23:25:51.028112 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:25:51.028081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51a95d7c_e171_4201_9d62_8dca8054e498.slice/crio-ddf9bc3522a7c68765673c72f9060e304e4b6c36a3d77a08e30cb0155dcb6e4e WatchSource:0}: Error finding container ddf9bc3522a7c68765673c72f9060e304e4b6c36a3d77a08e30cb0155dcb6e4e: Status 404 returned error can't find the container with id ddf9bc3522a7c68765673c72f9060e304e4b6c36a3d77a08e30cb0155dcb6e4e
Sep 17 23:25:51.103437 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:51.103406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj" event={"ID":"51a95d7c-e171-4201-9d62-8dca8054e498","Type":"ContainerStarted","Data":"ddf9bc3522a7c68765673c72f9060e304e4b6c36a3d77a08e30cb0155dcb6e4e"}
Sep 17 23:25:52.112116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:52.112022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj" event={"ID":"51a95d7c-e171-4201-9d62-8dca8054e498","Type":"ContainerStarted","Data":"efe59bbaeeafd48e672e037681ded66630448d6fcaaf6384145283333af2e7c5"}
Sep 17 23:25:52.132891 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:52.132835    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj" podStartSLOduration=2.132818002 podStartE2EDuration="2.132818002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:52.13150923 +0000 UTC m=+1162.561369077" watchObservedRunningTime="2026-09-17 23:25:52.132818002 +0000 UTC m=+1162.562677812"
Sep 17 23:25:53.751563 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:25:53.751530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:00.750060 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:00.750020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:26:00.750525 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:00.750076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:26:00.762468 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:00.762439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:26:00.829015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:00.828983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:01.170482 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:01.170452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:26:01.744143 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:01.744109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:02.365346 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:02.365312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:03.080749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:03.080705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:03.847824 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:03.847765    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:03.989189 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:03.989132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:04.544677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:04.544614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:05.497559 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:05.497508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:06.184355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:06.184305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:09.208609 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:09.208514    2580 generic.go:356] "Generic (PLEG): container finished" podID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerID="de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f" exitCode=0
Sep 17 23:26:09.208609 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:09.208574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" event={"ID":"32b101bd-be25-4f15-b585-5f45a64b1dcd","Type":"ContainerDied","Data":"de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f"}
Sep 17 23:26:09.209082 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:09.209024    2580 scope.go:117] "RemoveContainer" containerID="de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f"
Sep 17 23:26:09.210219 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:09.210195    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:26:09.210279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:09.210261    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:26:09.210319 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:09.210291    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:26:10.216614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:10.216580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" event={"ID":"32b101bd-be25-4f15-b585-5f45a64b1dcd","Type":"ContainerStarted","Data":"150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b"}
Sep 17 23:26:10.231217 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:10.231189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:26:10.231439 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:10.231416    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 17 23:26:10.231499 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:10.231477    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 17 23:26:11.231530 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:11.231493    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 17 23:26:11.231969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:11.231551    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 17 23:26:12.231596 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:12.231561    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 17 23:26:12.232020 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:12.231625    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 17 23:26:13.231080 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:13.231032    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 17 23:26:13.231261 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:13.231102    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 17 23:26:14.232072 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:14.232030    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 17 23:26:14.232466 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:14.232107    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 17 23:26:15.231060 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:15.231019    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 17 23:26:15.231246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:15.231078    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 17 23:26:16.235721 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:16.235688    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:26:16.236148 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:16.236006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:26:16.236702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:16.236685    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:26:27.484342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.484308    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"]
Sep 17 23:26:27.487978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.487957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.491258 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.491236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ab633\"/\"distinct-w4-7ab633-kserve-self-signed-certs\""
Sep 17 23:26:27.507061 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.507029    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"]
Sep 17 23:26:27.542786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.542748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:27.573129 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.573091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-model-cache\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.573310 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.573140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997wh\" (UniqueName: \"kubernetes.io/projected/1809a149-bd3c-4497-84e3-308e9015968b-kube-api-access-997wh\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.573310 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.573213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-home\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.573310 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.573271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1809a149-bd3c-4497-84e3-308e9015968b-tls-certs\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.573413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.573341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-dshm\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.573413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.573402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-tmp-dir\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674343 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1809a149-bd3c-4497-84e3-308e9015968b-tls-certs\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674524 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-dshm\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674524 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-tmp-dir\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674524 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-model-cache\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674524 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997wh\" (UniqueName: \"kubernetes.io/projected/1809a149-bd3c-4497-84e3-308e9015968b-kube-api-access-997wh\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674524 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-home\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-model-cache\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.674931 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-home\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.675085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.674977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-tmp-dir\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.676918 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.676897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-dshm\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.677212 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.677196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1809a149-bd3c-4497-84e3-308e9015968b-tls-certs\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.683387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.683357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-997wh\" (UniqueName: \"kubernetes.io/projected/1809a149-bd3c-4497-84e3-308e9015968b-kube-api-access-997wh\") pod \"distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:27.797131 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:27.797040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:28.071312 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:28.071284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"]
Sep 17 23:26:28.073076 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:26:28.073044    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1809a149_bd3c_4497_84e3_308e9015968b.slice/crio-50b30e6772bddd2fa00cfabb86f80292d0dd243e022d60b937e166c8e23d61ee WatchSource:0}: Error finding container 50b30e6772bddd2fa00cfabb86f80292d0dd243e022d60b937e166c8e23d61ee: Status 404 returned error can't find the container with id 50b30e6772bddd2fa00cfabb86f80292d0dd243e022d60b937e166c8e23d61ee
Sep 17 23:26:28.313658 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:28.313614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q" event={"ID":"1809a149-bd3c-4497-84e3-308e9015968b","Type":"ContainerStarted","Data":"990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9"}
Sep 17 23:26:28.313859 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:28.313664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q" event={"ID":"1809a149-bd3c-4497-84e3-308e9015968b","Type":"ContainerStarted","Data":"50b30e6772bddd2fa00cfabb86f80292d0dd243e022d60b937e166c8e23d61ee"}
Sep 17 23:26:28.334108 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:28.333995    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q" podStartSLOduration=1.333976585 podStartE2EDuration="1.333976585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:28.332034509 +0000 UTC m=+1198.761894328" watchObservedRunningTime="2026-09-17 23:26:28.333976585 +0000 UTC m=+1198.763836450"
Sep 17 23:26:30.097674 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:30.097647    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:26:30.098140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:30.098124    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:26:30.181388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:30.181355    2580 scope.go:117] "RemoveContainer" containerID="e75af2c1ae7d9c1a6255b5884f91b4634a4f114d9fad8b59479f01a51dc19ce2"
Sep 17 23:26:37.798104 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:37.798064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:37.798638 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:37.798119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:37.810957 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:37.810932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:38.377773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:38.377745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:26:46.463198 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:46.463162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:54.579436 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:26:54.579403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:01.567994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.567960    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"]
Sep 17 23:27:01.571844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.571821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"
Sep 17 23:27:01.574009 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.573991    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:27:01.574096 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.574066    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:27:01.578901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.578878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"]
Sep 17 23:27:01.617151 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.617118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b96d223-5a37-4c86-8436-3ffb8f97890c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6824f488-fqkdp\" (UID: \"3b96d223-5a37-4c86-8436-3ffb8f97890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"
Sep 17 23:27:01.718107 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.718076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b96d223-5a37-4c86-8436-3ffb8f97890c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6824f488-fqkdp\" (UID: \"3b96d223-5a37-4c86-8436-3ffb8f97890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"
Sep 17 23:27:01.718748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.718721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b96d223-5a37-4c86-8436-3ffb8f97890c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6824f488-fqkdp\" (UID: \"3b96d223-5a37-4c86-8436-3ffb8f97890c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"
Sep 17 23:27:01.838766 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.838684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:01.884850 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:01.884791    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"
Sep 17 23:27:02.158661 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:02.158636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"]
Sep 17 23:27:02.160687 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:27:02.160658    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b96d223_5a37_4c86_8436_3ffb8f97890c.slice/crio-f5aaf89bb002cdd3e5a86d3284c8c4a2611c17c531666ce50e667856c2e51cec WatchSource:0}: Error finding container f5aaf89bb002cdd3e5a86d3284c8c4a2611c17c531666ce50e667856c2e51cec: Status 404 returned error can't find the container with id f5aaf89bb002cdd3e5a86d3284c8c4a2611c17c531666ce50e667856c2e51cec
Sep 17 23:27:02.501119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:02.501091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp" event={"ID":"3b96d223-5a37-4c86-8436-3ffb8f97890c","Type":"ContainerStarted","Data":"f5aaf89bb002cdd3e5a86d3284c8c4a2611c17c531666ce50e667856c2e51cec"}
Sep 17 23:27:02.660017 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:02.659983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:03.332717 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:03.332636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:03.507959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:03.507923    2580 generic.go:356] "Generic (PLEG): container finished" podID="3b96d223-5a37-4c86-8436-3ffb8f97890c" containerID="fd168e7fa321dad6dc3e715b6608319c77d7f11b90cbfd34b08d73fc9baa1f12" exitCode=0
Sep 17 23:27:03.508124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:03.508010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp" event={"ID":"3b96d223-5a37-4c86-8436-3ffb8f97890c","Type":"ContainerDied","Data":"fd168e7fa321dad6dc3e715b6608319c77d7f11b90cbfd34b08d73fc9baa1f12"}
Sep 17 23:27:03.967919 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:03.967884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:04.653096 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:04.653072    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"
Sep 17 23:27:04.744320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:04.744292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:04.748508 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:04.748485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b96d223-5a37-4c86-8436-3ffb8f97890c-maas-api-service-ca\") pod \"3b96d223-5a37-4c86-8436-3ffb8f97890c\" (UID: \"3b96d223-5a37-4c86-8436-3ffb8f97890c\") "
Sep 17 23:27:04.748987 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:04.748959    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b96d223-5a37-4c86-8436-3ffb8f97890c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b96d223-5a37-4c86-8436-3ffb8f97890c" (UID: "3b96d223-5a37-4c86-8436-3ffb8f97890c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:04.850032 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:04.849938    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b96d223-5a37-4c86-8436-3ffb8f97890c-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:05.487792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:05.487761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:05.520983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:05.520940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp" event={"ID":"3b96d223-5a37-4c86-8436-3ffb8f97890c","Type":"ContainerDied","Data":"f5aaf89bb002cdd3e5a86d3284c8c4a2611c17c531666ce50e667856c2e51cec"}
Sep 17 23:27:05.520983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:05.520986    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"
Sep 17 23:27:05.521198 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:05.521004    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5aaf89bb002cdd3e5a86d3284c8c4a2611c17c531666ce50e667856c2e51cec"
Sep 17 23:27:06.250693 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:06.250659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:06.913093 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:06.913062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:08.482591 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:08.482553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.150671 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.150631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.164152 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.164119    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"]
Sep 17 23:27:17.164791 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.164754    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7" podUID="4cf6c118-b82e-4b78-823e-d9395e34ed40" containerName="payload-processing" containerID="cri-o://a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd" gracePeriod=30
Sep 17 23:27:17.248403 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.248367    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.358942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.358902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.427742 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.427715    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:27:17.429274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.429251    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"]
Sep 17 23:27:17.435535 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.435513    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6824f488-fqkdp"]
Sep 17 23:27:17.449404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.449354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.586090 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.586057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf6c118-b82e-4b78-823e-d9395e34ed40-plugins-config-volume\") pod \"4cf6c118-b82e-4b78-823e-d9395e34ed40\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") "
Sep 17 23:27:17.586267 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.586181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpx64\" (UniqueName: \"kubernetes.io/projected/4cf6c118-b82e-4b78-823e-d9395e34ed40-kube-api-access-dpx64\") pod \"4cf6c118-b82e-4b78-823e-d9395e34ed40\" (UID: \"4cf6c118-b82e-4b78-823e-d9395e34ed40\") "
Sep 17 23:27:17.586462 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.586440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf6c118-b82e-4b78-823e-d9395e34ed40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cf6c118-b82e-4b78-823e-d9395e34ed40" (UID: "4cf6c118-b82e-4b78-823e-d9395e34ed40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:17.586559 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.586545    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf6c118-b82e-4b78-823e-d9395e34ed40-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:17.588924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.588896    2580 generic.go:356] "Generic (PLEG): container finished" podID="4cf6c118-b82e-4b78-823e-d9395e34ed40" containerID="a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd" exitCode=0
Sep 17 23:27:17.589054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.588905    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf6c118-b82e-4b78-823e-d9395e34ed40-kube-api-access-dpx64" (OuterVolumeSpecName: "kube-api-access-dpx64") pod "4cf6c118-b82e-4b78-823e-d9395e34ed40" (UID: "4cf6c118-b82e-4b78-823e-d9395e34ed40"). InnerVolumeSpecName "kube-api-access-dpx64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:17.589054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.588933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7" event={"ID":"4cf6c118-b82e-4b78-823e-d9395e34ed40","Type":"ContainerDied","Data":"a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd"}
Sep 17 23:27:17.589177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.589064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7" event={"ID":"4cf6c118-b82e-4b78-823e-d9395e34ed40","Type":"ContainerDied","Data":"388d632fd67d67697e7801034229a1a016123a0c54e789509f2af4bfbe3753f1"}
Sep 17 23:27:17.589177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.588947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"
Sep 17 23:27:17.589177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.589100    2580 scope.go:117] "RemoveContainer" containerID="a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd"
Sep 17 23:27:17.605366 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.605343    2580 scope.go:117] "RemoveContainer" containerID="a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd"
Sep 17 23:27:17.605645 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:27:17.605622    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd\": container with ID starting with a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd not found: ID does not exist" containerID="a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd"
Sep 17 23:27:17.605723 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.605657    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd"} err="failed to get container status \"a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd\": rpc error: code = NotFound desc = could not find container \"a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd\": container with ID starting with a63725274e7cb00e355c419937cbe2d0a2a26a855c84efaffc4b68332a5b89fd not found: ID does not exist"
Sep 17 23:27:17.616335 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.616302    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"]
Sep 17 23:27:17.620177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.620155    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6824f488-847dd44849-cpgg7"]
Sep 17 23:27:17.687760 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.687665    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpx64\" (UniqueName: \"kubernetes.io/projected/4cf6c118-b82e-4b78-823e-d9395e34ed40-kube-api-access-dpx64\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:17.778921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:17.778885    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:18.110234 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:18.110149    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b96d223-5a37-4c86-8436-3ffb8f97890c" path="/var/lib/kubelet/pods/3b96d223-5a37-4c86-8436-3ffb8f97890c/volumes"
Sep 17 23:27:18.110562 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:18.110547    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf6c118-b82e-4b78-823e-d9395e34ed40" path="/var/lib/kubelet/pods/4cf6c118-b82e-4b78-823e-d9395e34ed40/volumes"
Sep 17 23:27:18.724569 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:18.724538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:19.515220 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:19.515187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:20.326604 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:20.326570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:20.557276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:20.557216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:21.163079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:21.163040    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:21.983838 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:21.983788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:22.737177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:22.737147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:25.532777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:25.532740    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"]
Sep 17 23:27:25.583562 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:25.583533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:34.505009 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:34.504969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:40.552653 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.552598    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj" podUID="51a95d7c-e171-4201-9d62-8dca8054e498" containerName="main" containerID="cri-o://efe59bbaeeafd48e672e037681ded66630448d6fcaaf6384145283333af2e7c5" gracePeriod=45
Sep 17 23:27:40.728168 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.727867    2580 generic.go:356] "Generic (PLEG): container finished" podID="51a95d7c-e171-4201-9d62-8dca8054e498" containerID="efe59bbaeeafd48e672e037681ded66630448d6fcaaf6384145283333af2e7c5" exitCode=0
Sep 17 23:27:40.728168 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.728018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj" event={"ID":"51a95d7c-e171-4201-9d62-8dca8054e498","Type":"ContainerDied","Data":"efe59bbaeeafd48e672e037681ded66630448d6fcaaf6384145283333af2e7c5"}
Sep 17 23:27:40.806525 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.806460    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:27:40.925578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-home\") pod \"51a95d7c-e171-4201-9d62-8dca8054e498\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") "
Sep 17 23:27:40.925769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-tmp-dir\") pod \"51a95d7c-e171-4201-9d62-8dca8054e498\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") "
Sep 17 23:27:40.925769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925655    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a95d7c-e171-4201-9d62-8dca8054e498-tls-certs\") pod \"51a95d7c-e171-4201-9d62-8dca8054e498\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") "
Sep 17 23:27:40.925769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925682    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-dshm\") pod \"51a95d7c-e171-4201-9d62-8dca8054e498\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") "
Sep 17 23:27:40.925769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925725    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-model-cache\") pod \"51a95d7c-e171-4201-9d62-8dca8054e498\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") "
Sep 17 23:27:40.925769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l5dd\" (UniqueName: \"kubernetes.io/projected/51a95d7c-e171-4201-9d62-8dca8054e498-kube-api-access-4l5dd\") pod \"51a95d7c-e171-4201-9d62-8dca8054e498\" (UID: \"51a95d7c-e171-4201-9d62-8dca8054e498\") "
Sep 17 23:27:40.926063 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-home" (OuterVolumeSpecName: "home") pod "51a95d7c-e171-4201-9d62-8dca8054e498" (UID: "51a95d7c-e171-4201-9d62-8dca8054e498"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:40.926063 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.925974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51a95d7c-e171-4201-9d62-8dca8054e498" (UID: "51a95d7c-e171-4201-9d62-8dca8054e498"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:40.926063 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.926042    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-model-cache" (OuterVolumeSpecName: "model-cache") pod "51a95d7c-e171-4201-9d62-8dca8054e498" (UID: "51a95d7c-e171-4201-9d62-8dca8054e498"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:40.926205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.926142    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:40.926205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.926174    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:40.926205 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.926185    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:40.928069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.928041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-dshm" (OuterVolumeSpecName: "dshm") pod "51a95d7c-e171-4201-9d62-8dca8054e498" (UID: "51a95d7c-e171-4201-9d62-8dca8054e498"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:40.928149 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.928131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a95d7c-e171-4201-9d62-8dca8054e498-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51a95d7c-e171-4201-9d62-8dca8054e498" (UID: "51a95d7c-e171-4201-9d62-8dca8054e498"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:40.928218 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:40.928199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a95d7c-e171-4201-9d62-8dca8054e498-kube-api-access-4l5dd" (OuterVolumeSpecName: "kube-api-access-4l5dd") pod "51a95d7c-e171-4201-9d62-8dca8054e498" (UID: "51a95d7c-e171-4201-9d62-8dca8054e498"). InnerVolumeSpecName "kube-api-access-4l5dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:41.027502 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.027471    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a95d7c-e171-4201-9d62-8dca8054e498-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:41.027502 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.027495    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a95d7c-e171-4201-9d62-8dca8054e498-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:41.027502 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.027504    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l5dd\" (UniqueName: \"kubernetes.io/projected/51a95d7c-e171-4201-9d62-8dca8054e498-kube-api-access-4l5dd\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:27:41.736333 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.736304    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"
Sep 17 23:27:41.736752 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.736344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj" event={"ID":"51a95d7c-e171-4201-9d62-8dca8054e498","Type":"ContainerDied","Data":"ddf9bc3522a7c68765673c72f9060e304e4b6c36a3d77a08e30cb0155dcb6e4e"}
Sep 17 23:27:41.736752 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.736398    2580 scope.go:117] "RemoveContainer" containerID="efe59bbaeeafd48e672e037681ded66630448d6fcaaf6384145283333af2e7c5"
Sep 17 23:27:41.758177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.758151    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"]
Sep 17 23:27:41.761975 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:41.761951    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6824f488/e2e-lifecycle-model-6824f488-kserve-55c5c77d59-f7xkj"]
Sep 17 23:27:42.109991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:42.109909    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a95d7c-e171-4201-9d62-8dca8054e498" path="/var/lib/kubelet/pods/51a95d7c-e171-4201-9d62-8dca8054e498/volumes"
Sep 17 23:27:54.040750 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.040708    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"]
Sep 17 23:27:54.041352 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041332    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf6c118-b82e-4b78-823e-d9395e34ed40" containerName="payload-processing"
Sep 17 23:27:54.041411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041357    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf6c118-b82e-4b78-823e-d9395e34ed40" containerName="payload-processing"
Sep 17 23:27:54.041411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041370    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b96d223-5a37-4c86-8436-3ffb8f97890c" containerName="revoke-keys"
Sep 17 23:27:54.041411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041378    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b96d223-5a37-4c86-8436-3ffb8f97890c" containerName="revoke-keys"
Sep 17 23:27:54.041411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041407    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51a95d7c-e171-4201-9d62-8dca8054e498" containerName="main"
Sep 17 23:27:54.041538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041415    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51a95d7c-e171-4201-9d62-8dca8054e498" containerName="main"
Sep 17 23:27:54.041538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041520    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51a95d7c-e171-4201-9d62-8dca8054e498" containerName="main"
Sep 17 23:27:54.041538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041533    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf6c118-b82e-4b78-823e-d9395e34ed40" containerName="payload-processing"
Sep 17 23:27:54.041629 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.041542    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b96d223-5a37-4c86-8436-3ffb8f97890c" containerName="revoke-keys"
Sep 17 23:27:54.045635 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.045618    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.048310 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.048278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ab633\"/\"distinct-2-w4-7ab633-kserve-self-signed-certs\""
Sep 17 23:27:54.057301 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.057276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"]
Sep 17 23:27:54.058792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.058770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:54.167881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.167845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwjd\" (UniqueName: \"kubernetes.io/projected/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-kube-api-access-nwwjd\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.167881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.167889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tls-certs\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.168124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.167907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-home\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.168124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.167931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-model-cache\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.168124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.168019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-dshm\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.168124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.168042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tmp-dir\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.268588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.268548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-dshm\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.268783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.268601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tmp-dir\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.268783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.268672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwjd\" (UniqueName: \"kubernetes.io/projected/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-kube-api-access-nwwjd\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.268783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.268701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tls-certs\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.268783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.268723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-home\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.268783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.268752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-model-cache\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.269199 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.269172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-model-cache\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.269295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.269279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-home\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.269340 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.269282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tmp-dir\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.271112 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.271086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-dshm\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.271355 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.271336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tls-certs\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.276441 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.276418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwjd\" (UniqueName: \"kubernetes.io/projected/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-kube-api-access-nwwjd\") pod \"distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") " pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.357232 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.357146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:27:54.631513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.631491    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"]
Sep 17 23:27:54.633947 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:27:54.633920    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ebdb1e5_d445_4cda_bfd0_54b8138a8363.slice/crio-ebd301d4613a8fc38d8ec9054a29d3bb9b959eb1cfb71abb08c15b00a5ea3185 WatchSource:0}: Error finding container ebd301d4613a8fc38d8ec9054a29d3bb9b959eb1cfb71abb08c15b00a5ea3185: Status 404 returned error can't find the container with id ebd301d4613a8fc38d8ec9054a29d3bb9b959eb1cfb71abb08c15b00a5ea3185
Sep 17 23:27:54.816143 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.816095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr" event={"ID":"1ebdb1e5-d445-4cda-bfd0-54b8138a8363","Type":"ContainerStarted","Data":"3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086"}
Sep 17 23:27:54.816143 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.816140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr" event={"ID":"1ebdb1e5-d445-4cda-bfd0-54b8138a8363","Type":"ContainerStarted","Data":"ebd301d4613a8fc38d8ec9054a29d3bb9b959eb1cfb71abb08c15b00a5ea3185"}
Sep 17 23:27:54.832459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:27:54.832411    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr" podStartSLOduration=0.832395432 podStartE2EDuration="832.395432ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:54.831507899 +0000 UTC m=+1285.261367719" watchObservedRunningTime="2026-09-17 23:27:54.832395432 +0000 UTC m=+1285.262255249"
Sep 17 23:28:01.211599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:01.211567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:02.178591 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.178557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:02.424087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.424044    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"]
Sep 17 23:28:02.427753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.427729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.430883 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.430818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"kube-root-ca.crt\""
Sep 17 23:28:02.430883 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.430853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"openshift-service-ca.crt\""
Sep 17 23:28:02.431080 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.430828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"auth-test-model-6275e9-kserve-self-signed-certs\""
Sep 17 23:28:02.431080 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.430830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"default-dockercfg-vtrc6\""
Sep 17 23:28:02.437502 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.437478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"]
Sep 17 23:28:02.455383 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.455347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-home\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.455566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.455403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt47h\" (UniqueName: \"kubernetes.io/projected/0fdff86f-117e-40dc-bf45-0b56bc2654b7-kube-api-access-qt47h\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.455566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.455510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-dshm\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.455566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.455537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tmp-dir\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.455688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.455574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-model-cache\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.455688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.455660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tls-certs\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.556709 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.556665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt47h\" (UniqueName: \"kubernetes.io/projected/0fdff86f-117e-40dc-bf45-0b56bc2654b7-kube-api-access-qt47h\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.556951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.556756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-dshm\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.556951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.556779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tmp-dir\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.556951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.556830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-model-cache\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.556951 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.556901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tls-certs\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.557188 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.556959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-home\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.557337 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.557314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-model-cache\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.557406 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.557340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tmp-dir\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.557406 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.557353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-home\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.559281 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.559259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-dshm\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.559530 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.559513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tls-certs\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.565239 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.565218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt47h\" (UniqueName: \"kubernetes.io/projected/0fdff86f-117e-40dc-bf45-0b56bc2654b7-kube-api-access-qt47h\") pod \"auth-test-model-6275e9-kserve-795cdfbb99-mnqdc\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.738081 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.737990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:02.830332 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.830225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:02.964720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:02.964688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:03.016987 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:03.016962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"]
Sep 17 23:28:03.019789 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:28:03.019754    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fdff86f_117e_40dc_bf45_0b56bc2654b7.slice/crio-7c6997b8c4505c045e4f53be5b86576e30dcc54ff2daffed703f09e7d0782638 WatchSource:0}: Error finding container 7c6997b8c4505c045e4f53be5b86576e30dcc54ff2daffed703f09e7d0782638: Status 404 returned error can't find the container with id 7c6997b8c4505c045e4f53be5b86576e30dcc54ff2daffed703f09e7d0782638
Sep 17 23:28:03.872848 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:03.872811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc" event={"ID":"0fdff86f-117e-40dc-bf45-0b56bc2654b7","Type":"ContainerStarted","Data":"2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d"}
Sep 17 23:28:03.873232 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:03.872858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc" event={"ID":"0fdff86f-117e-40dc-bf45-0b56bc2654b7","Type":"ContainerStarted","Data":"7c6997b8c4505c045e4f53be5b86576e30dcc54ff2daffed703f09e7d0782638"}
Sep 17 23:28:03.893312 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:03.893255    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc" podStartSLOduration=1.8932361979999999 podStartE2EDuration="1.893236198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:03.890897964 +0000 UTC m=+1294.320757781" watchObservedRunningTime="2026-09-17 23:28:03.893236198 +0000 UTC m=+1294.323096017"
Sep 17 23:28:04.358018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:04.357969    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:28:04.358018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:04.358029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:28:04.370675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:04.370644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:28:04.890293 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:04.890263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:28:08.560685 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:08.560649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:12.738563 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:12.738527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:12.739029 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:12.738585    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:12.751496 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:12.751471    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:12.932295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:12.932265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:28:14.168067 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:14.168030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:14.247248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:14.247192    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:14.422904 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:14.422791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:15.061734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:15.061695    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:15.693513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:15.693459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:15.794764 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:15.794726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:15.908986 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:15.908944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:16.484294 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:16.484259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:16.584282 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:16.584251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:16.684263 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:16.684225    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:17.205106 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:17.205072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:17.930238 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:17.930201    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:23.900830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:23.900778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:23.969469 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:23.969432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:24.548254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:24.548216    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:30.381056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:30.381020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:30.483921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:30.483840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:30.583456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:30.583414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:31.166187 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:31.166153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:41.071734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:41.071698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:41.295695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:41.295659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:42.134288 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:42.134253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:43.007507 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:28:43.007473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:02.204233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:02.204146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:08.474777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:08.474734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:08.574786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:08.574752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:08.810083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:08.809995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:09.099277 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:09.099188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:09.413612 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:09.413578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:11.754830 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:11.754777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:11.937741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:11.937708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:12.029719 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:12.029637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:12.656721 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:12.656689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:13.414070 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:13.414029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:14.077673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:14.077639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:14.842971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:14.842935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:15.479703 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:15.479667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:16.153657 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:16.153621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:16.254547 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:16.254511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:16.354992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:16.354958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:16.613577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:16.613487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:16.719120 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:16.719086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:17.825878 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:17.825784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:18.550990 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:18.550954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:40.554606 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:40.554566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:41.435786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:41.435755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:41.975159 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:41.975122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:42.245224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:42.245136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:42.413994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:42.413962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:42.714055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:42.714023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:42.946946 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:42.946911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:43.047720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:43.047635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:43.602583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:43.602555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:44.352624 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:44.352589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:45.024015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:45.023982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:45.745012 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:45.744977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:46.216943 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.216907    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:46.318514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.318472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:46.415621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.415559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:46.611028 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.610934    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"]
Sep 17 23:29:46.614658 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.614641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"
Sep 17 23:29:46.617329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.617306    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:29:46.617468 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.617450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:29:46.623556 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.623531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"]
Sep 17 23:29:46.727200 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.727166    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:46.741442 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.741410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fb95dc-da4d-4533-b1f8-e87a753af0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4\" (UID: \"14fb95dc-da4d-4533-b1f8-e87a753af0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"
Sep 17 23:29:46.842458 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.842417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fb95dc-da4d-4533-b1f8-e87a753af0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4\" (UID: \"14fb95dc-da4d-4533-b1f8-e87a753af0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"
Sep 17 23:29:46.843119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.843099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fb95dc-da4d-4533-b1f8-e87a753af0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4\" (UID: \"14fb95dc-da4d-4533-b1f8-e87a753af0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"
Sep 17 23:29:46.926773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:46.926738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"
Sep 17 23:29:47.201322 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:47.201293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"]
Sep 17 23:29:47.203431 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:29:47.203400    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fb95dc_da4d_4533_b1f8_e87a753af0a3.slice/crio-9a0619d2c34071201ebfcf573bbe85f73c606e90679f0205bb1caebb3a6ab48c WatchSource:0}: Error finding container 9a0619d2c34071201ebfcf573bbe85f73c606e90679f0205bb1caebb3a6ab48c: Status 404 returned error can't find the container with id 9a0619d2c34071201ebfcf573bbe85f73c606e90679f0205bb1caebb3a6ab48c
Sep 17 23:29:47.447256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:47.447220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4" event={"ID":"14fb95dc-da4d-4533-b1f8-e87a753af0a3","Type":"ContainerStarted","Data":"9a0619d2c34071201ebfcf573bbe85f73c606e90679f0205bb1caebb3a6ab48c"}
Sep 17 23:29:47.663689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:47.663645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:48.349152 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:48.349116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:48.454343 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:48.454299    2580 generic.go:356] "Generic (PLEG): container finished" podID="14fb95dc-da4d-4533-b1f8-e87a753af0a3" containerID="9781f82069cc7de23a912717a89a755e6659676375bbaf51a352cffcd8d07202" exitCode=0
Sep 17 23:29:48.454508 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:48.454377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4" event={"ID":"14fb95dc-da4d-4533-b1f8-e87a753af0a3","Type":"ContainerDied","Data":"9781f82069cc7de23a912717a89a755e6659676375bbaf51a352cffcd8d07202"}
Sep 17 23:29:49.036247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:49.036218    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:49.599060 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:49.599033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"
Sep 17 23:29:49.670921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:49.670885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fb95dc-da4d-4533-b1f8-e87a753af0a3-maas-api-service-ca\") pod \"14fb95dc-da4d-4533-b1f8-e87a753af0a3\" (UID: \"14fb95dc-da4d-4533-b1f8-e87a753af0a3\") "
Sep 17 23:29:49.671261 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:49.671238    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fb95dc-da4d-4533-b1f8-e87a753af0a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14fb95dc-da4d-4533-b1f8-e87a753af0a3" (UID: "14fb95dc-da4d-4533-b1f8-e87a753af0a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:29:49.727016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:49.726923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:49.772099 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:49.772064    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fb95dc-da4d-4533-b1f8-e87a753af0a3-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:29:50.466153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:50.466121    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"
Sep 17 23:29:50.466153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:50.466129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4" event={"ID":"14fb95dc-da4d-4533-b1f8-e87a753af0a3","Type":"ContainerDied","Data":"9a0619d2c34071201ebfcf573bbe85f73c606e90679f0205bb1caebb3a6ab48c"}
Sep 17 23:29:50.466374 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:50.466184    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a0619d2c34071201ebfcf573bbe85f73c606e90679f0205bb1caebb3a6ab48c"
Sep 17 23:29:50.543610 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:50.543577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:51.257480 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:51.257444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:51.416359 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:51.416321    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"]
Sep 17 23:29:51.425783 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:51.425754    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"]
Sep 17 23:29:51.488752 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:51.488718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:52.129445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:52.129412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:52.759136 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:52.759105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:52.860015 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:52.859979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:52.999636 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:52.999598    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:53.204037 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.204009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:53.303535 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.303501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:53.309396 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.309364    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"]
Sep 17 23:29:53.309775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.309759    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fb95dc-da4d-4533-b1f8-e87a753af0a3" containerName="revoke-keys"
Sep 17 23:29:53.309834 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.309776    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fb95dc-da4d-4533-b1f8-e87a753af0a3" containerName="revoke-keys"
Sep 17 23:29:53.309875 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.309868    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fb95dc-da4d-4533-b1f8-e87a753af0a3" containerName="revoke-keys"
Sep 17 23:29:53.313070 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.313054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.315278 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.315252    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-00d34b\""
Sep 17 23:29:53.315423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.315359    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-00d34b-dockercfg-nr8c4\""
Sep 17 23:29:53.322580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.322558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"]
Sep 17 23:29:53.349261 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.349229    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"]
Sep 17 23:29:53.352813 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.352786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.364146 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.364112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"]
Sep 17 23:29:53.410658 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.410622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a76e18-92f3-4403-a18d-be5cc9566084-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.410903 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.410680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-plugins-config-volume\") pod \"payload-processing-e2e-api-b-00d34b-54f9774875-rc782\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") " pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.410903 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.410780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxr7z\" (UniqueName: \"kubernetes.io/projected/71a76e18-92f3-4403-a18d-be5cc9566084-kube-api-access-sxr7z\") pod \"payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.411063 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.410965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrvz\" (UniqueName: \"kubernetes.io/projected/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-kube-api-access-pxrvz\") pod \"payload-processing-e2e-api-b-00d34b-54f9774875-rc782\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") " pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.511960 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.511863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxr7z\" (UniqueName: \"kubernetes.io/projected/71a76e18-92f3-4403-a18d-be5cc9566084-kube-api-access-sxr7z\") pod \"payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.511960 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.511954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrvz\" (UniqueName: \"kubernetes.io/projected/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-kube-api-access-pxrvz\") pod \"payload-processing-e2e-api-b-00d34b-54f9774875-rc782\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") " pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.512206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.512005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a76e18-92f3-4403-a18d-be5cc9566084-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.512206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.512040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-plugins-config-volume\") pod \"payload-processing-e2e-api-b-00d34b-54f9774875-rc782\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") " pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.512673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.512654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-plugins-config-volume\") pod \"payload-processing-e2e-api-b-00d34b-54f9774875-rc782\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") " pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.512743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.512723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a76e18-92f3-4403-a18d-be5cc9566084-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.522609 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.522584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrvz\" (UniqueName: \"kubernetes.io/projected/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-kube-api-access-pxrvz\") pod \"payload-processing-e2e-api-b-00d34b-54f9774875-rc782\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") " pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.522741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.522583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxr7z\" (UniqueName: \"kubernetes.io/projected/71a76e18-92f3-4403-a18d-be5cc9566084-kube-api-access-sxr7z\") pod \"payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.625135 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.625102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:53.634383 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.634341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:53.665523 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.665474    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:53.907549 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.907523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"]
Sep 17 23:29:53.909567 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:29:53.909539    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a76e18_92f3_4403_a18d_be5cc9566084.slice/crio-ea333d5a2f22d779b3ddb835584bda5ab2713d378e8cd2ec082e399544cd49bf WatchSource:0}: Error finding container ea333d5a2f22d779b3ddb835584bda5ab2713d378e8cd2ec082e399544cd49bf: Status 404 returned error can't find the container with id ea333d5a2f22d779b3ddb835584bda5ab2713d378e8cd2ec082e399544cd49bf
Sep 17 23:29:53.959144 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:53.959120    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"]
Sep 17 23:29:53.960939 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:29:53.960914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf3ebab_2d3c_4f1d_b6df_1943eebe8986.slice/crio-f6e91e7486ab2bdff6c4d70d369809aea67464c15e7b6abba20de8e7a074f41b WatchSource:0}: Error finding container f6e91e7486ab2bdff6c4d70d369809aea67464c15e7b6abba20de8e7a074f41b: Status 404 returned error can't find the container with id f6e91e7486ab2bdff6c4d70d369809aea67464c15e7b6abba20de8e7a074f41b
Sep 17 23:29:54.440840 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.440788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:54.491398 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.491358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782" event={"ID":"abf3ebab-2d3c-4f1d-b6df-1943eebe8986","Type":"ContainerStarted","Data":"07bdd40c9ab8633556eea628e74a1b3bf8e12a77f7697d372eb0854af33d7c48"}
Sep 17 23:29:54.491614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.491427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782" event={"ID":"abf3ebab-2d3c-4f1d-b6df-1943eebe8986","Type":"ContainerStarted","Data":"f6e91e7486ab2bdff6c4d70d369809aea67464c15e7b6abba20de8e7a074f41b"}
Sep 17 23:29:54.491614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.491570    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:29:54.493193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.493163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr" event={"ID":"71a76e18-92f3-4403-a18d-be5cc9566084","Type":"ContainerStarted","Data":"8e56735f21760b1afdd909b29ed7160922f28911f01d61bc1510ef081c6d5f77"}
Sep 17 23:29:54.493351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.493214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr" event={"ID":"71a76e18-92f3-4403-a18d-be5cc9566084","Type":"ContainerStarted","Data":"ea333d5a2f22d779b3ddb835584bda5ab2713d378e8cd2ec082e399544cd49bf"}
Sep 17 23:29:54.493351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.493272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:29:54.510044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.510003    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782" podStartSLOduration=1.509988304 podStartE2EDuration="1.509988304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:54.508561647 +0000 UTC m=+1404.938421476" watchObservedRunningTime="2026-09-17 23:29:54.509988304 +0000 UTC m=+1404.939848183"
Sep 17 23:29:54.525732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:54.525668    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr" podStartSLOduration=1.525648741 podStartE2EDuration="1.525648741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:54.524642602 +0000 UTC m=+1404.954502433" watchObservedRunningTime="2026-09-17 23:29:54.525648741 +0000 UTC m=+1404.955508559"
Sep 17 23:29:56.507254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:56.507220    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerID="3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92" exitCode=137
Sep 17 23:29:56.507622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:56.507278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" event={"ID":"ad5da990-f291-48f6-91dd-96b54e91d5c3","Type":"ContainerDied","Data":"3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92"}
Sep 17 23:29:56.507622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:56.507615    2580 scope.go:117] "RemoveContainer" containerID="3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92"
Sep 17 23:29:56.763969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:56.763889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:57.518427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:57.518394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" event={"ID":"ad5da990-f291-48f6-91dd-96b54e91d5c3","Type":"ContainerStarted","Data":"5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158"}
Sep 17 23:29:57.518894 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:29:57.518606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:30:00.149381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.149340    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"]
Sep 17 23:30:00.152915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.152891    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"]
Sep 17 23:30:00.153059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.153007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"
Sep 17 23:30:00.155947 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.155929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"
Sep 17 23:30:00.160109 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.160089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:30:00.163635 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.163619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"
Sep 17 23:30:00.170023 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.170001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"
Sep 17 23:30:00.175233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.175210    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"]
Sep 17 23:30:00.178900 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.178874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"
Sep 17 23:30:00.200107 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.200070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"]
Sep 17 23:30:00.219043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.218787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"
Sep 17 23:30:00.220073 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.220040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"]
Sep 17 23:30:00.228320 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.225446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"]
Sep 17 23:30:00.256811 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.256751    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"]
Sep 17 23:30:00.266208 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.266180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"
Sep 17 23:30:00.274230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.274179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"]
Sep 17 23:30:00.281424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.281379    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"
Sep 17 23:30:00.454138 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.454112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"]
Sep 17 23:30:00.456192 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:30:00.456166    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a4f5ee_f1c7_4e2d_8343_8a8bd2dc5274.slice/crio-f25d9f9edbe4948f9beea8b3f7367445b7dfe9d5baee40d806ca2c14f98e6913 WatchSource:0}: Error finding container f25d9f9edbe4948f9beea8b3f7367445b7dfe9d5baee40d806ca2c14f98e6913: Status 404 returned error can't find the container with id f25d9f9edbe4948f9beea8b3f7367445b7dfe9d5baee40d806ca2c14f98e6913
Sep 17 23:30:00.470542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.470522    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"]
Sep 17 23:30:00.472220 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:30:00.472190    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1004eb63_7572_4d03_94fb_0fb092855ab5.slice/crio-6fdc70b8c2839f8c6187f1657a7084bb7b07de228343cda7db1bea6ccf7a4688 WatchSource:0}: Error finding container 6fdc70b8c2839f8c6187f1657a7084bb7b07de228343cda7db1bea6ccf7a4688: Status 404 returned error can't find the container with id 6fdc70b8c2839f8c6187f1657a7084bb7b07de228343cda7db1bea6ccf7a4688
Sep 17 23:30:00.531038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.531009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"]
Sep 17 23:30:00.534391 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:30:00.534364    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fac102_be59_466a_8bee_6d52778d4ee3.slice/crio-e1b398a4b0a3cf13b3f631fcb3de903a4f1faad18c584a3f1b029522c8a1704c WatchSource:0}: Error finding container e1b398a4b0a3cf13b3f631fcb3de903a4f1faad18c584a3f1b029522c8a1704c: Status 404 returned error can't find the container with id e1b398a4b0a3cf13b3f631fcb3de903a4f1faad18c584a3f1b029522c8a1704c
Sep 17 23:30:00.536491 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.536461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp" event={"ID":"1004eb63-7572-4d03-94fb-0fb092855ab5","Type":"ContainerStarted","Data":"6fdc70b8c2839f8c6187f1657a7084bb7b07de228343cda7db1bea6ccf7a4688"}
Sep 17 23:30:00.537736 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.537715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x" event={"ID":"02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274","Type":"ContainerStarted","Data":"f25d9f9edbe4948f9beea8b3f7367445b7dfe9d5baee40d806ca2c14f98e6913"}
Sep 17 23:30:00.580410 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:00.580387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"]
Sep 17 23:30:00.582495 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:30:00.582456    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0b925b_23a3_45c7_a37f_ece41272acd7.slice/crio-9ce3b5c6dcd2d752a027d23856c3f4bf7855531a59c090ac34cde243e3a11123 WatchSource:0}: Error finding container 9ce3b5c6dcd2d752a027d23856c3f4bf7855531a59c090ac34cde243e3a11123: Status 404 returned error can't find the container with id 9ce3b5c6dcd2d752a027d23856c3f4bf7855531a59c090ac34cde243e3a11123
Sep 17 23:30:01.546864 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.546827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz" event={"ID":"64fac102-be59-466a-8bee-6d52778d4ee3","Type":"ContainerStarted","Data":"28f21ba693315aff421fc4521fafd5e9cd7ad37d1c247a92dee2cd4ed9a199f1"}
Sep 17 23:30:01.547308 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.546927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz" event={"ID":"64fac102-be59-466a-8bee-6d52778d4ee3","Type":"ContainerStarted","Data":"e1b398a4b0a3cf13b3f631fcb3de903a4f1faad18c584a3f1b029522c8a1704c"}
Sep 17 23:30:01.548584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.548558    2580 generic.go:356] "Generic (PLEG): container finished" podID="02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274" containerID="284cfde79198c16ef8fa074d05279e4a9d8e17512768518d2cfc6db4f3f6d1c9" exitCode=0
Sep 17 23:30:01.548695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.548634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x" event={"ID":"02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274","Type":"ContainerDied","Data":"284cfde79198c16ef8fa074d05279e4a9d8e17512768518d2cfc6db4f3f6d1c9"}
Sep 17 23:30:01.550294 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.550252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp" event={"ID":"1004eb63-7572-4d03-94fb-0fb092855ab5","Type":"ContainerStarted","Data":"d6d2cc4c0f982f3b79e440818a3654e8ce274700ee345cc42127610787111cb0"}
Sep 17 23:30:01.551791 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.551768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67" event={"ID":"7f0b925b-23a3-45c7-a37f-ece41272acd7","Type":"ContainerStarted","Data":"6fbad6055323a2d041397e68d0cb2b1a57eaf4464bff941a6a6793cd792bd5d3"}
Sep 17 23:30:01.551905 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.551822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67" event={"ID":"7f0b925b-23a3-45c7-a37f-ece41272acd7","Type":"ContainerStarted","Data":"9ce3b5c6dcd2d752a027d23856c3f4bf7855531a59c090ac34cde243e3a11123"}
Sep 17 23:30:01.562370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.562331    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz" podStartSLOduration=1.56231757 podStartE2EDuration="1.56231757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.560998806 +0000 UTC m=+1411.990858656" watchObservedRunningTime="2026-09-17 23:30:01.56231757 +0000 UTC m=+1411.992177390"
Sep 17 23:30:01.592826 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.592765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp" podStartSLOduration=1.5927474130000001 podStartE2EDuration="1.592747413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.590657041 +0000 UTC m=+1412.020516860" watchObservedRunningTime="2026-09-17 23:30:01.592747413 +0000 UTC m=+1412.022607231"
Sep 17 23:30:01.606648 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:01.606601    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67" podStartSLOduration=1.606585207 podStartE2EDuration="1.606585207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.604751561 +0000 UTC m=+1412.034611382" watchObservedRunningTime="2026-09-17 23:30:01.606585207 +0000 UTC m=+1412.036445068"
Sep 17 23:30:02.560698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.560665    2580 generic.go:356] "Generic (PLEG): container finished" podID="1004eb63-7572-4d03-94fb-0fb092855ab5" containerID="d6d2cc4c0f982f3b79e440818a3654e8ce274700ee345cc42127610787111cb0" exitCode=0
Sep 17 23:30:02.561147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.560757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp" event={"ID":"1004eb63-7572-4d03-94fb-0fb092855ab5","Type":"ContainerDied","Data":"d6d2cc4c0f982f3b79e440818a3654e8ce274700ee345cc42127610787111cb0"}
Sep 17 23:30:02.562529 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.562504    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f0b925b-23a3-45c7-a37f-ece41272acd7" containerID="6fbad6055323a2d041397e68d0cb2b1a57eaf4464bff941a6a6793cd792bd5d3" exitCode=0
Sep 17 23:30:02.562654 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.562629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67" event={"ID":"7f0b925b-23a3-45c7-a37f-ece41272acd7","Type":"ContainerDied","Data":"6fbad6055323a2d041397e68d0cb2b1a57eaf4464bff941a6a6793cd792bd5d3"}
Sep 17 23:30:02.564176 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.564155    2580 generic.go:356] "Generic (PLEG): container finished" podID="64fac102-be59-466a-8bee-6d52778d4ee3" containerID="28f21ba693315aff421fc4521fafd5e9cd7ad37d1c247a92dee2cd4ed9a199f1" exitCode=0
Sep 17 23:30:02.564282 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.564217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz" event={"ID":"64fac102-be59-466a-8bee-6d52778d4ee3","Type":"ContainerDied","Data":"28f21ba693315aff421fc4521fafd5e9cd7ad37d1c247a92dee2cd4ed9a199f1"}
Sep 17 23:30:02.662583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.662551    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"]
Sep 17 23:30:02.683013 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.682970    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"]
Sep 17 23:30:02.683223 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.683196    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf" podUID="05076899-68ae-4e95-8a3c-a347efb2d140" containerName="payload-processing" containerID="cri-o://2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7" gracePeriod=30
Sep 17 23:30:02.737382 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.737354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:02.813342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.813276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"
Sep 17 23:30:02.835560 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.835513    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:02.855334 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.855298    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"]
Sep 17 23:30:02.862574 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.862543    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-0b4d93-j9tm4"]
Sep 17 23:30:02.896044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.896002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:02.956755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:02.956729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:30:03.114773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.114685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05076899-68ae-4e95-8a3c-a347efb2d140-plugins-config-volume\") pod \"05076899-68ae-4e95-8a3c-a347efb2d140\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") "
Sep 17 23:30:03.114773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.114737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq87d\" (UniqueName: \"kubernetes.io/projected/05076899-68ae-4e95-8a3c-a347efb2d140-kube-api-access-dq87d\") pod \"05076899-68ae-4e95-8a3c-a347efb2d140\" (UID: \"05076899-68ae-4e95-8a3c-a347efb2d140\") "
Sep 17 23:30:03.115093 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.115069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05076899-68ae-4e95-8a3c-a347efb2d140-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05076899-68ae-4e95-8a3c-a347efb2d140" (UID: "05076899-68ae-4e95-8a3c-a347efb2d140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:03.117201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.117174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05076899-68ae-4e95-8a3c-a347efb2d140-kube-api-access-dq87d" (OuterVolumeSpecName: "kube-api-access-dq87d") pod "05076899-68ae-4e95-8a3c-a347efb2d140" (UID: "05076899-68ae-4e95-8a3c-a347efb2d140"). InnerVolumeSpecName "kube-api-access-dq87d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:03.216302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.216270    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05076899-68ae-4e95-8a3c-a347efb2d140-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:03.216302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.216299    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq87d\" (UniqueName: \"kubernetes.io/projected/05076899-68ae-4e95-8a3c-a347efb2d140-kube-api-access-dq87d\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:03.233849 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.233794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:03.319885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.319846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:03.571560 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.571529    2580 generic.go:356] "Generic (PLEG): container finished" podID="05076899-68ae-4e95-8a3c-a347efb2d140" containerID="2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7" exitCode=0
Sep 17 23:30:03.572083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.571580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"
Sep 17 23:30:03.572083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.571618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf" event={"ID":"05076899-68ae-4e95-8a3c-a347efb2d140","Type":"ContainerDied","Data":"2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7"}
Sep 17 23:30:03.572083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.571678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf" event={"ID":"05076899-68ae-4e95-8a3c-a347efb2d140","Type":"ContainerDied","Data":"cd6f37a0cb1b19e225a0472920458f53e1f8a2403a443441bec3a31656bdb408"}
Sep 17 23:30:03.572083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.571715    2580 scope.go:117] "RemoveContainer" containerID="2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7"
Sep 17 23:30:03.573445 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.573409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x" event={"ID":"02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274","Type":"ContainerDied","Data":"f25d9f9edbe4948f9beea8b3f7367445b7dfe9d5baee40d806ca2c14f98e6913"}
Sep 17 23:30:03.573537 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.573463    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f25d9f9edbe4948f9beea8b3f7367445b7dfe9d5baee40d806ca2c14f98e6913"
Sep 17 23:30:03.573927 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.573899    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"
Sep 17 23:30:03.584340 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.584050    2580 scope.go:117] "RemoveContainer" containerID="2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7"
Sep 17 23:30:03.584415 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:30:03.584378    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7\": container with ID starting with 2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7 not found: ID does not exist" containerID="2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7"
Sep 17 23:30:03.584482 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.584409    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7"} err="failed to get container status \"2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7\": rpc error: code = NotFound desc = could not find container \"2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7\": container with ID starting with 2e54ba7d1337faebbf1222f4c925a1d089ffdb0d6286310efc9681d7105263e7 not found: ID does not exist"
Sep 17 23:30:03.611678 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.611642    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"]
Sep 17 23:30:03.618221 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.618120    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0b4d93-6749f94779-j6kqf"]
Sep 17 23:30:03.792873 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.792847    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"
Sep 17 23:30:03.794897 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.794876    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"
Sep 17 23:30:03.798473 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:03.798457    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"
Sep 17 23:30:04.110250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.110215    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05076899-68ae-4e95-8a3c-a347efb2d140" path="/var/lib/kubelet/pods/05076899-68ae-4e95-8a3c-a347efb2d140/volumes"
Sep 17 23:30:04.110628 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.110613    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14fb95dc-da4d-4533-b1f8-e87a753af0a3" path="/var/lib/kubelet/pods/14fb95dc-da4d-4533-b1f8-e87a753af0a3/volumes"
Sep 17 23:30:04.175231 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.175199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:04.581370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.581344    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"
Sep 17 23:30:04.581847 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.581375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz" event={"ID":"64fac102-be59-466a-8bee-6d52778d4ee3","Type":"ContainerDied","Data":"e1b398a4b0a3cf13b3f631fcb3de903a4f1faad18c584a3f1b029522c8a1704c"}
Sep 17 23:30:04.581847 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.581428    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1b398a4b0a3cf13b3f631fcb3de903a4f1faad18c584a3f1b029522c8a1704c"
Sep 17 23:30:04.582762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.582743    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"
Sep 17 23:30:04.582762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.582754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp" event={"ID":"1004eb63-7572-4d03-94fb-0fb092855ab5","Type":"ContainerDied","Data":"6fdc70b8c2839f8c6187f1657a7084bb7b07de228343cda7db1bea6ccf7a4688"}
Sep 17 23:30:04.582960 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.582823    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fdc70b8c2839f8c6187f1657a7084bb7b07de228343cda7db1bea6ccf7a4688"
Sep 17 23:30:04.584269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.584238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67" event={"ID":"7f0b925b-23a3-45c7-a37f-ece41272acd7","Type":"ContainerDied","Data":"9ce3b5c6dcd2d752a027d23856c3f4bf7855531a59c090ac34cde243e3a11123"}
Sep 17 23:30:04.584269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.584265    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"
Sep 17 23:30:04.584404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.584281    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ce3b5c6dcd2d752a027d23856c3f4bf7855531a59c090ac34cde243e3a11123"
Sep 17 23:30:04.613517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.613488    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"]
Sep 17 23:30:04.618005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.617979    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0b4d93-29828130-r2xtz"]
Sep 17 23:30:04.739908 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:04.739877    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:05.501623 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:05.501591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:30:05.501822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:05.501659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:30:05.548773 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:05.548732    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:06.109394 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.109362    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fac102-be59-466a-8bee-6d52778d4ee3" path="/var/lib/kubelet/pods/64fac102-be59-466a-8bee-6d52778d4ee3/volumes"
Sep 17 23:30:06.348365 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.348335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:06.436719 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.436681    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d" podUID="6e0133aa-a194-47ca-8a9a-99c570817943" containerName="main" containerID="cri-o://6c66f00f798ea96f7a4958aeb16eb27f6d75bffcde4c447ec4efcf9c08f6a83d" gracePeriod=45
Sep 17 23:30:06.448246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.448213    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8" podUID="7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" containerName="main" containerID="cri-o://f8de253b99580e8009e16df59d650fe55dae494cd906c50b70a6f9a67125ffb1" gracePeriod=45
Sep 17 23:30:06.598105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.598074    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e0133aa-a194-47ca-8a9a-99c570817943" containerID="6c66f00f798ea96f7a4958aeb16eb27f6d75bffcde4c447ec4efcf9c08f6a83d" exitCode=0
Sep 17 23:30:06.598292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.598193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d" event={"ID":"6e0133aa-a194-47ca-8a9a-99c570817943","Type":"ContainerDied","Data":"6c66f00f798ea96f7a4958aeb16eb27f6d75bffcde4c447ec4efcf9c08f6a83d"}
Sep 17 23:30:06.599736 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.599705    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" containerID="f8de253b99580e8009e16df59d650fe55dae494cd906c50b70a6f9a67125ffb1" exitCode=0
Sep 17 23:30:06.599912 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.599756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8" event={"ID":"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0","Type":"ContainerDied","Data":"f8de253b99580e8009e16df59d650fe55dae494cd906c50b70a6f9a67125ffb1"}
Sep 17 23:30:06.732365 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.732327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:30:06.739352 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.739322    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:30:06.857054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-model-cache\") pod \"6e0133aa-a194-47ca-8a9a-99c570817943\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") "
Sep 17 23:30:06.857245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tls-certs\") pod \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") "
Sep 17 23:30:06.857245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-dshm\") pod \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") "
Sep 17 23:30:06.857245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-dshm\") pod \"6e0133aa-a194-47ca-8a9a-99c570817943\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") "
Sep 17 23:30:06.857245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-home\") pod \"6e0133aa-a194-47ca-8a9a-99c570817943\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") "
Sep 17 23:30:06.857245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwcxd\" (UniqueName: \"kubernetes.io/projected/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-kube-api-access-cwcxd\") pod \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") "
Sep 17 23:30:06.857245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-model-cache\") pod \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") "
Sep 17 23:30:06.857245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj768\" (UniqueName: \"kubernetes.io/projected/6e0133aa-a194-47ca-8a9a-99c570817943-kube-api-access-fj768\") pod \"6e0133aa-a194-47ca-8a9a-99c570817943\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") "
Sep 17 23:30:06.857600 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857253    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-tmp-dir\") pod \"6e0133aa-a194-47ca-8a9a-99c570817943\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") "
Sep 17 23:30:06.857600 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e0133aa-a194-47ca-8a9a-99c570817943" (UID: "6e0133aa-a194-47ca-8a9a-99c570817943"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.857600 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0133aa-a194-47ca-8a9a-99c570817943-tls-certs\") pod \"6e0133aa-a194-47ca-8a9a-99c570817943\" (UID: \"6e0133aa-a194-47ca-8a9a-99c570817943\") "
Sep 17 23:30:06.857600 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tmp-dir\") pod \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") "
Sep 17 23:30:06.857600 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857503    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-home\") pod \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\" (UID: \"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0\") "
Sep 17 23:30:06.857895 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-home" (OuterVolumeSpecName: "home") pod "6e0133aa-a194-47ca-8a9a-99c570817943" (UID: "6e0133aa-a194-47ca-8a9a-99c570817943"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.857970 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.857940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" (UID: "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.858083 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.858065    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.858181 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.858161    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" (UID: "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.858279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.858181    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-home" (OuterVolumeSpecName: "home") pod "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" (UID: "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.858279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.858233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e0133aa-a194-47ca-8a9a-99c570817943" (UID: "6e0133aa-a194-47ca-8a9a-99c570817943"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.860181 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.860151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-kube-api-access-cwcxd" (OuterVolumeSpecName: "kube-api-access-cwcxd") pod "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" (UID: "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0"). InnerVolumeSpecName "kube-api-access-cwcxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:06.860479 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.860455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0133aa-a194-47ca-8a9a-99c570817943-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e0133aa-a194-47ca-8a9a-99c570817943" (UID: "6e0133aa-a194-47ca-8a9a-99c570817943"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:30:06.860479 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.860473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-dshm" (OuterVolumeSpecName: "dshm") pod "6e0133aa-a194-47ca-8a9a-99c570817943" (UID: "6e0133aa-a194-47ca-8a9a-99c570817943"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.860641 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.860455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-dshm" (OuterVolumeSpecName: "dshm") pod "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" (UID: "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:06.860641 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.860473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" (UID: "7ac41bd9-8645-4d0f-b023-5c7fd42e71e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:30:06.860641 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.860565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0133aa-a194-47ca-8a9a-99c570817943-kube-api-access-fj768" (OuterVolumeSpecName: "kube-api-access-fj768") pod "6e0133aa-a194-47ca-8a9a-99c570817943" (UID: "6e0133aa-a194-47ca-8a9a-99c570817943"). InnerVolumeSpecName "kube-api-access-fj768". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959616    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959649    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959660    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwcxd\" (UniqueName: \"kubernetes.io/projected/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-kube-api-access-cwcxd\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959671    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959682    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj768\" (UniqueName: \"kubernetes.io/projected/6e0133aa-a194-47ca-8a9a-99c570817943-kube-api-access-fj768\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959691    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e0133aa-a194-47ca-8a9a-99c570817943-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959699    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e0133aa-a194-47ca-8a9a-99c570817943-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959705    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959712    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959719    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:06.959720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:06.959726    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:07.134985 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.134952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:07.607098 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.607066    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"
Sep 17 23:30:07.607294 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.607101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d" event={"ID":"6e0133aa-a194-47ca-8a9a-99c570817943","Type":"ContainerDied","Data":"8fc87c3bd924dbf7dca316c773eb139f2158fb969f781cbfd88d77161fd850c2"}
Sep 17 23:30:07.607294 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.607154    2580 scope.go:117] "RemoveContainer" containerID="6c66f00f798ea96f7a4958aeb16eb27f6d75bffcde4c447ec4efcf9c08f6a83d"
Sep 17 23:30:07.608673 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.608653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8" event={"ID":"7ac41bd9-8645-4d0f-b023-5c7fd42e71e0","Type":"ContainerDied","Data":"171f602298904a99d62f4a6036da812dded7c644b0155f17845fb1cf68a95d3f"}
Sep 17 23:30:07.608762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.608673    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"
Sep 17 23:30:07.617594 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.617570    2580 scope.go:117] "RemoveContainer" containerID="f8de253b99580e8009e16df59d650fe55dae494cd906c50b70a6f9a67125ffb1"
Sep 17 23:30:07.630922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.630900    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"]
Sep 17 23:30:07.636437 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.636415    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/premium-w0-0b4d93-kserve-b5657bf54-27w6d"]
Sep 17 23:30:07.648189 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.648164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"]
Sep 17 23:30:07.650087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.650067    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0b4d93/simulator-w0-0b4d93-kserve-96f48887-h9bl8"]
Sep 17 23:30:07.832309 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:07.832212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:08.110010 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:08.109927    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e0133aa-a194-47ca-8a9a-99c570817943" path="/var/lib/kubelet/pods/6e0133aa-a194-47ca-8a9a-99c570817943/volumes"
Sep 17 23:30:08.110331 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:08.110318    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" path="/var/lib/kubelet/pods/7ac41bd9-8645-4d0f-b023-5c7fd42e71e0/volumes"
Sep 17 23:30:08.481098 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:08.481066    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:08.525503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:08.525474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:30:09.544256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:09.544221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:09.644218 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:09.644185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:09.811594 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:09.811505    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"]
Sep 17 23:30:09.811770 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:09.811745    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy" containerID="cri-o://f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae" gracePeriod=30
Sep 17 23:30:09.828857 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:09.828779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:10.004285 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:10.004250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:10.802615 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:10.802581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:11.097963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:11.097885    2580 patch_prober.go:28] interesting pod/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 23:30:11.097963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:11.097952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:30:11.151013 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:11.150963    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:11.513991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:11.513954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:12.189120 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:12.189087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:12.833546 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:12.833498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:13.491441 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:13.491406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:14.503087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:14.503048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:15.069366 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.069338    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:30:15.139576 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-certs\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.139755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-credential-socket\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.139755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139613    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-token\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.139755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-data\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.139755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-socket\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.139755 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-podinfo\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.140047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6nxs\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-kube-api-access-b6nxs\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.140047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139864    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-envoy\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.140047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istiod-ca-cert\") pod \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\" (UID: \"27c76884-ca63-40e3-b4d0-9a7f9a71d691\") "
Sep 17 23:30:15.140047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:15.140047 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.139984    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:15.140249 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.140049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:15.140329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.140309    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.140387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.140327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:15.140387 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.140335    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-credential-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.140483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.140398    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-workload-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.140483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.140404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-data" (OuterVolumeSpecName: "istio-data") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:15.142107 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.142076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-token" (OuterVolumeSpecName: "istio-token") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:15.142273 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.142248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 23:30:15.142316 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.142253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-kube-api-access-b6nxs" (OuterVolumeSpecName: "kube-api-access-b6nxs") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "kube-api-access-b6nxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:15.142398 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.142384    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "27c76884-ca63-40e3-b4d0-9a7f9a71d691" (UID: "27c76884-ca63-40e3-b4d0-9a7f9a71d691"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:15.241694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.241658    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-envoy\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.241694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.241692    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istiod-ca-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.241694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.241702    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-token\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.241694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.241709    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-data\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.241694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.241718    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c76884-ca63-40e3-b4d0-9a7f9a71d691-istio-podinfo\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.242046 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.241726    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6nxs\" (UniqueName: \"kubernetes.io/projected/27c76884-ca63-40e3-b4d0-9a7f9a71d691-kube-api-access-b6nxs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:15.245625 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.245598    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:15.663490 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.663457    2580 generic.go:356] "Generic (PLEG): container finished" podID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerID="f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae" exitCode=0
Sep 17 23:30:15.663975 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.663543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" event={"ID":"27c76884-ca63-40e3-b4d0-9a7f9a71d691","Type":"ContainerDied","Data":"f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae"}
Sep 17 23:30:15.663975 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.663587    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"
Sep 17 23:30:15.663975 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.663603    2580 scope.go:117] "RemoveContainer" containerID="f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae"
Sep 17 23:30:15.663975 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.663591    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh" event={"ID":"27c76884-ca63-40e3-b4d0-9a7f9a71d691","Type":"ContainerDied","Data":"47e5e286a8f9f44b0cd64635e827abde657710de1fe4dfa4869ab1d4b587945a"}
Sep 17 23:30:15.673900 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.673882    2580 scope.go:117] "RemoveContainer" containerID="ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71"
Sep 17 23:30:15.683224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.683198    2580 scope.go:117] "RemoveContainer" containerID="f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae"
Sep 17 23:30:15.683606 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:30:15.683581    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae\": container with ID starting with f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae not found: ID does not exist" containerID="f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae"
Sep 17 23:30:15.683700 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.683620    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae"} err="failed to get container status \"f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae\": rpc error: code = NotFound desc = could not find container \"f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae\": container with ID starting with f2bbb86584ba99d3c0b45a95d1687012e220bd6049c227a8bca9d259659b45ae not found: ID does not exist"
Sep 17 23:30:15.683700 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.683645    2580 scope.go:117] "RemoveContainer" containerID="ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71"
Sep 17 23:30:15.683973 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:30:15.683954    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71\": container with ID starting with ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71 not found: ID does not exist" containerID="ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71"
Sep 17 23:30:15.684028 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.683982    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71"} err="failed to get container status \"ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71\": rpc error: code = NotFound desc = could not find container \"ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71\": container with ID starting with ef279a037a3c8a4b9ad35ee68c9d5e91b12f1c8f1f8e766193951a29a3ae5e71 not found: ID does not exist"
Sep 17 23:30:15.692655 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.692619    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"]
Sep 17 23:30:15.703421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.703390    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0b4d93-openshift-default-76c65cf6bd-fvgqh"]
Sep 17 23:30:15.985378 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:15.985282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:16.108877 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:16.108842    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" path="/var/lib/kubelet/pods/27c76884-ca63-40e3-b4d0-9a7f9a71d691/volumes"
Sep 17 23:30:16.872720 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:16.872674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:17.640413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:17.640380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:18.455337 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:18.455302    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:19.127978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:19.127943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:20.219868 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:20.219751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:20.930717 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:20.930655    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:24.139916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:24.139832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:26.372904 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:26.372856    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:28.636718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:28.636641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:28.767942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:28.767912    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:28.829443 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:28.829387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:29.353751 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:29.353714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:29.717725 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:29.717686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:29.817151 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:29.817122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:29.937775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:29.937736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:30.120457 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:30.120362    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:30.276934 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:30.276898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:30.453055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:30.453028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:31.426279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:31.426246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:32.143186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:32.143157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:32.928328 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:32.928290    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:33.745151 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:33.745114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:34.483447 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:34.483408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:35.211919 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:35.211888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:35.628968 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:35.628872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:36.415985 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:36.415952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:36.489352 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:36.489312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:36.687819 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:36.687710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:37.381642 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:37.381603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:38.335203 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:38.335158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:39.014706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:39.014674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:39.831615 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:39.831585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:40.838172 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:40.838142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:41.522758 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:41.522724    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:42.345180 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:42.345146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:43.141639 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:43.141601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:47.262213 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:47.262178    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:47.319875 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:47.319837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:49.832819 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:49.832760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.625512 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.625480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.654110 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.654069    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"]
Sep 17 23:30:52.654356 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.654322    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782" podUID="abf3ebab-2d3c-4f1d-b6df-1943eebe8986" containerName="payload-processing" containerID="cri-o://07bdd40c9ab8633556eea628e74a1b3bf8e12a77f7697d372eb0854af33d7c48" gracePeriod=30
Sep 17 23:30:52.663902 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.663875    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"]
Sep 17 23:30:52.664334 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.664300    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr" podUID="71a76e18-92f3-4403-a18d-be5cc9566084" containerName="payload-pre-processing" containerID="cri-o://8e56735f21760b1afdd909b29ed7160922f28911f01d61bc1510ef081c6d5f77" gracePeriod=30
Sep 17 23:30:52.704759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.704723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.773721 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.773690    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.844811 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.844763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.899031 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.898988    2580 generic.go:356] "Generic (PLEG): container finished" podID="abf3ebab-2d3c-4f1d-b6df-1943eebe8986" containerID="07bdd40c9ab8633556eea628e74a1b3bf8e12a77f7697d372eb0854af33d7c48" exitCode=0
Sep 17 23:30:52.899235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.899060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782" event={"ID":"abf3ebab-2d3c-4f1d-b6df-1943eebe8986","Type":"ContainerDied","Data":"07bdd40c9ab8633556eea628e74a1b3bf8e12a77f7697d372eb0854af33d7c48"}
Sep 17 23:30:52.900944 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.900921    2580 generic.go:356] "Generic (PLEG): container finished" podID="71a76e18-92f3-4403-a18d-be5cc9566084" containerID="8e56735f21760b1afdd909b29ed7160922f28911f01d61bc1510ef081c6d5f77" exitCode=0
Sep 17 23:30:52.901068 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.900989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr" event={"ID":"71a76e18-92f3-4403-a18d-be5cc9566084","Type":"ContainerDied","Data":"8e56735f21760b1afdd909b29ed7160922f28911f01d61bc1510ef081c6d5f77"}
Sep 17 23:30:52.930394 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.930370    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:30:52.942936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:52.942915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:30:53.017106 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.017069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a76e18-92f3-4403-a18d-be5cc9566084-plugins-config-volume\") pod \"71a76e18-92f3-4403-a18d-be5cc9566084\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") "
Sep 17 23:30:53.017106 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.017111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxrvz\" (UniqueName: \"kubernetes.io/projected/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-kube-api-access-pxrvz\") pod \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") "
Sep 17 23:30:53.017368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.017215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-plugins-config-volume\") pod \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\" (UID: \"abf3ebab-2d3c-4f1d-b6df-1943eebe8986\") "
Sep 17 23:30:53.017368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.017287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxr7z\" (UniqueName: \"kubernetes.io/projected/71a76e18-92f3-4403-a18d-be5cc9566084-kube-api-access-sxr7z\") pod \"71a76e18-92f3-4403-a18d-be5cc9566084\" (UID: \"71a76e18-92f3-4403-a18d-be5cc9566084\") "
Sep 17 23:30:53.017567 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.017534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a76e18-92f3-4403-a18d-be5cc9566084-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71a76e18-92f3-4403-a18d-be5cc9566084" (UID: "71a76e18-92f3-4403-a18d-be5cc9566084"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:53.017684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.017612    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71a76e18-92f3-4403-a18d-be5cc9566084-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:53.017684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.017616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abf3ebab-2d3c-4f1d-b6df-1943eebe8986" (UID: "abf3ebab-2d3c-4f1d-b6df-1943eebe8986"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:53.019443 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.019419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a76e18-92f3-4403-a18d-be5cc9566084-kube-api-access-sxr7z" (OuterVolumeSpecName: "kube-api-access-sxr7z") pod "71a76e18-92f3-4403-a18d-be5cc9566084" (UID: "71a76e18-92f3-4403-a18d-be5cc9566084"). InnerVolumeSpecName "kube-api-access-sxr7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:53.019525 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.019483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-kube-api-access-pxrvz" (OuterVolumeSpecName: "kube-api-access-pxrvz") pod "abf3ebab-2d3c-4f1d-b6df-1943eebe8986" (UID: "abf3ebab-2d3c-4f1d-b6df-1943eebe8986"). InnerVolumeSpecName "kube-api-access-pxrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:53.118978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.118933    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:53.118978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.118970    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxr7z\" (UniqueName: \"kubernetes.io/projected/71a76e18-92f3-4403-a18d-be5cc9566084-kube-api-access-sxr7z\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:53.118978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.118980    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxrvz\" (UniqueName: \"kubernetes.io/projected/abf3ebab-2d3c-4f1d-b6df-1943eebe8986-kube-api-access-pxrvz\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:30:53.188349 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.188257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:53.303194 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.303162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:53.909239 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.909206    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"
Sep 17 23:30:53.909239 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.909208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782" event={"ID":"abf3ebab-2d3c-4f1d-b6df-1943eebe8986","Type":"ContainerDied","Data":"f6e91e7486ab2bdff6c4d70d369809aea67464c15e7b6abba20de8e7a074f41b"}
Sep 17 23:30:53.909735 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.909301    2580 scope.go:117] "RemoveContainer" containerID="07bdd40c9ab8633556eea628e74a1b3bf8e12a77f7697d372eb0854af33d7c48"
Sep 17 23:30:53.911077 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.911054    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"
Sep 17 23:30:53.911195 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.911072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr" event={"ID":"71a76e18-92f3-4403-a18d-be5cc9566084","Type":"ContainerDied","Data":"ea333d5a2f22d779b3ddb835584bda5ab2713d378e8cd2ec082e399544cd49bf"}
Sep 17 23:30:53.921022 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.920994    2580 scope.go:117] "RemoveContainer" containerID="8e56735f21760b1afdd909b29ed7160922f28911f01d61bc1510ef081c6d5f77"
Sep 17 23:30:53.937468 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.937443    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"]
Sep 17 23:30:53.946592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.946558    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-00d34b-54f9774875-rc782"]
Sep 17 23:30:53.948677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.948653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:53.959560 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.959536    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"]
Sep 17 23:30:53.966384 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:53.966359    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-00d34b-5cb64f478b-6clwr"]
Sep 17 23:30:54.109738 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:54.109703    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a76e18-92f3-4403-a18d-be5cc9566084" path="/var/lib/kubelet/pods/71a76e18-92f3-4403-a18d-be5cc9566084/volumes"
Sep 17 23:30:54.110181 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:54.110166    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf3ebab-2d3c-4f1d-b6df-1943eebe8986" path="/var/lib/kubelet/pods/abf3ebab-2d3c-4f1d-b6df-1943eebe8986/volumes"
Sep 17 23:30:54.866160 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:54.866121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:55.943400 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:55.943366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:56.738514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:56.738481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:57.450721 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:57.450689    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:59.043213 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:59.043180    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:59.143677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:59.143644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:59.304336 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:59.304246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:59.868706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:30:59.868676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:00.096710 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.096668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:00.210622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.210588    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"]
Sep 17 23:31:00.211109 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211088    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0133aa-a194-47ca-8a9a-99c570817943" containerName="main"
Sep 17 23:31:00.211109 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211107    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0133aa-a194-47ca-8a9a-99c570817943" containerName="main"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211123    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211130    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0b925b-23a3-45c7-a37f-ece41272acd7" containerName="cleanup"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211136    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0b925b-23a3-45c7-a37f-ece41272acd7" containerName="cleanup"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211144    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf3ebab-2d3c-4f1d-b6df-1943eebe8986" containerName="payload-processing"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211150    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf3ebab-2d3c-4f1d-b6df-1943eebe8986" containerName="payload-processing"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211178    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a76e18-92f3-4403-a18d-be5cc9566084" containerName="payload-pre-processing"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211184    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a76e18-92f3-4403-a18d-be5cc9566084" containerName="payload-pre-processing"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211196    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274" containerName="cleanup"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211202    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274" containerName="cleanup"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211209    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05076899-68ae-4e95-8a3c-a347efb2d140" containerName="payload-processing"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211214    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05076899-68ae-4e95-8a3c-a347efb2d140" containerName="payload-processing"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211220    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" containerName="main"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211226    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" containerName="main"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211232    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1004eb63-7572-4d03-94fb-0fb092855ab5" containerName="cleanup"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211236    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1004eb63-7572-4d03-94fb-0fb092855ab5" containerName="cleanup"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211249    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fac102-be59-466a-8bee-6d52778d4ee3" containerName="cleanup"
Sep 17 23:31:00.211254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211255    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fac102-be59-466a-8bee-6d52778d4ee3" containerName="cleanup"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211317    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211326    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fac102-be59-466a-8bee-6d52778d4ee3" containerName="cleanup"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211331    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0b925b-23a3-45c7-a37f-ece41272acd7" containerName="cleanup"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211339    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05076899-68ae-4e95-8a3c-a347efb2d140" containerName="payload-processing"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211343    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e0133aa-a194-47ca-8a9a-99c570817943" containerName="main"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211350    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a76e18-92f3-4403-a18d-be5cc9566084" containerName="payload-pre-processing"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211355    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1004eb63-7572-4d03-94fb-0fb092855ab5" containerName="cleanup"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211361    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac41bd9-8645-4d0f-b023-5c7fd42e71e0" containerName="main"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211365    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274" containerName="cleanup"
Sep 17 23:31:00.211790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.211371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf3ebab-2d3c-4f1d-b6df-1943eebe8986" containerName="payload-processing"
Sep 17 23:31:00.214457 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.214442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"
Sep 17 23:31:00.216919 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.216896    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:31:00.217044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.216981    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:31:00.221396 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.221373    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"]
Sep 17 23:31:00.292351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.292316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a6a01d-ea3b-4b11-bd63-d61fcfa3849c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx\" (UID: \"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"
Sep 17 23:31:00.393620 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.393575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a6a01d-ea3b-4b11-bd63-d61fcfa3849c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx\" (UID: \"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"
Sep 17 23:31:00.394221 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.394199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a6a01d-ea3b-4b11-bd63-d61fcfa3849c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx\" (UID: \"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"
Sep 17 23:31:00.526792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.526695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"
Sep 17 23:31:00.801276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.801090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"]
Sep 17 23:31:00.803434 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:31:00.803395    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a6a01d_ea3b_4b11_bd63_d61fcfa3849c.slice/crio-8b7f373f345e60e652aa8647ad128d0011dfda6442cc71d2c66ff97788b3a6fe WatchSource:0}: Error finding container 8b7f373f345e60e652aa8647ad128d0011dfda6442cc71d2c66ff97788b3a6fe: Status 404 returned error can't find the container with id 8b7f373f345e60e652aa8647ad128d0011dfda6442cc71d2c66ff97788b3a6fe
Sep 17 23:31:00.803548 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.803431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:00.952456 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:00.952421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx" event={"ID":"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c","Type":"ContainerStarted","Data":"8b7f373f345e60e652aa8647ad128d0011dfda6442cc71d2c66ff97788b3a6fe"}
Sep 17 23:31:01.641145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:01.641107    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:01.966164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:01.966127    2580 generic.go:356] "Generic (PLEG): container finished" podID="96a6a01d-ea3b-4b11-bd63-d61fcfa3849c" containerID="1bc697abed33d85f793e74ed292a3ae43a0e5e311e2da4a4b89461391234dd4b" exitCode=0
Sep 17 23:31:01.966376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:01.966218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx" event={"ID":"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c","Type":"ContainerDied","Data":"1bc697abed33d85f793e74ed292a3ae43a0e5e311e2da4a4b89461391234dd4b"}
Sep 17 23:31:02.461318 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:02.461284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:03.106169 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:03.106142    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"
Sep 17 23:31:03.223545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:03.223460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a6a01d-ea3b-4b11-bd63-d61fcfa3849c-maas-api-service-ca\") pod \"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c\" (UID: \"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c\") "
Sep 17 23:31:03.223894 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:03.223872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a6a01d-ea3b-4b11-bd63-d61fcfa3849c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96a6a01d-ea3b-4b11-bd63-d61fcfa3849c" (UID: "96a6a01d-ea3b-4b11-bd63-d61fcfa3849c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:03.324700 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:03.324666    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a6a01d-ea3b-4b11-bd63-d61fcfa3849c-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:03.978450 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:03.978420    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"
Sep 17 23:31:03.978659 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:03.978459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx" event={"ID":"96a6a01d-ea3b-4b11-bd63-d61fcfa3849c","Type":"ContainerDied","Data":"8b7f373f345e60e652aa8647ad128d0011dfda6442cc71d2c66ff97788b3a6fe"}
Sep 17 23:31:03.978659 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:03.978502    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7f373f345e60e652aa8647ad128d0011dfda6442cc71d2c66ff97788b3a6fe"
Sep 17 23:31:04.203516 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:04.203476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:04.304044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:04.303961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:04.754633 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:04.754596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:05.078175 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:05.078089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"]
Sep 17 23:31:05.083921 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:05.083493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"]
Sep 17 23:31:05.103922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:05.103859    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"]
Sep 17 23:31:05.162367 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:05.162322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:10.071072 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:10.071034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:10.144310 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:10.144275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:10.256626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:10.256589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:10.318708 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:10.318672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:10.611693 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:10.611663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:10.764907 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:10.764873    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:11.532538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:11.532501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:12.215677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.215640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:12.764304 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.764268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:12.787923 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.787885    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"]
Sep 17 23:31:12.788164 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.788138    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98" podUID="3973213c-acc3-4030-a510-19690f545bd6" containerName="payload-processing" containerID="cri-o://29bcadf7d2cbf70c41304de5a45d4344b1665fd7b9412bc926bfb4c9fdd2df2c" gracePeriod=30
Sep 17 23:31:12.799159 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.799022    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"]
Sep 17 23:31:12.799235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.799214    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb" podUID="a8d687b9-8e77-442e-8522-0532fd561fa0" containerName="payload-pre-processing" containerID="cri-o://fb1f60ef543b4efab67880ec7c8c3e040c7f1a2ff75b05b06ebdeaa6a85da713" gracePeriod=30
Sep 17 23:31:12.829079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.829022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:12.891176 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.891139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:12.986859 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.986825    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"]
Sep 17 23:31:12.988922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.988881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:12.990147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:12.990123    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ab633-vkxgx"]
Sep 17 23:31:13.033829 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.033780    2580 generic.go:356] "Generic (PLEG): container finished" podID="a8d687b9-8e77-442e-8522-0532fd561fa0" containerID="fb1f60ef543b4efab67880ec7c8c3e040c7f1a2ff75b05b06ebdeaa6a85da713" exitCode=0
Sep 17 23:31:13.033977 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.033846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb" event={"ID":"a8d687b9-8e77-442e-8522-0532fd561fa0","Type":"ContainerDied","Data":"fb1f60ef543b4efab67880ec7c8c3e040c7f1a2ff75b05b06ebdeaa6a85da713"}
Sep 17 23:31:13.036621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.036498    2580 generic.go:356] "Generic (PLEG): container finished" podID="3973213c-acc3-4030-a510-19690f545bd6" containerID="29bcadf7d2cbf70c41304de5a45d4344b1665fd7b9412bc926bfb4c9fdd2df2c" exitCode=0
Sep 17 23:31:13.036860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.036629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98" event={"ID":"3973213c-acc3-4030-a510-19690f545bd6","Type":"ContainerDied","Data":"29bcadf7d2cbf70c41304de5a45d4344b1665fd7b9412bc926bfb4c9fdd2df2c"}
Sep 17 23:31:13.036860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.036668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98" event={"ID":"3973213c-acc3-4030-a510-19690f545bd6","Type":"ContainerDied","Data":"fb77c71900391127568217ad2c14b86f5690a479c532098847fbed971d5f5178"}
Sep 17 23:31:13.036860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.036685    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb77c71900391127568217ad2c14b86f5690a479c532098847fbed971d5f5178"
Sep 17 23:31:13.047672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.047647    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:31:13.075013 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.074980    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:31:13.087195 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.087161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:13.129727 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.129694    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqg29\" (UniqueName: \"kubernetes.io/projected/3973213c-acc3-4030-a510-19690f545bd6-kube-api-access-pqg29\") pod \"3973213c-acc3-4030-a510-19690f545bd6\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") "
Sep 17 23:31:13.129919 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.129775    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973213c-acc3-4030-a510-19690f545bd6-plugins-config-volume\") pod \"3973213c-acc3-4030-a510-19690f545bd6\" (UID: \"3973213c-acc3-4030-a510-19690f545bd6\") "
Sep 17 23:31:13.130191 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.130157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3973213c-acc3-4030-a510-19690f545bd6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3973213c-acc3-4030-a510-19690f545bd6" (UID: "3973213c-acc3-4030-a510-19690f545bd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:13.130297 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.130230    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973213c-acc3-4030-a510-19690f545bd6-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:13.132005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.131982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3973213c-acc3-4030-a510-19690f545bd6-kube-api-access-pqg29" (OuterVolumeSpecName: "kube-api-access-pqg29") pod "3973213c-acc3-4030-a510-19690f545bd6" (UID: "3973213c-acc3-4030-a510-19690f545bd6"). InnerVolumeSpecName "kube-api-access-pqg29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:13.205520 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.205455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:13.231228 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.231201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d687b9-8e77-442e-8522-0532fd561fa0-plugins-config-volume\") pod \"a8d687b9-8e77-442e-8522-0532fd561fa0\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") "
Sep 17 23:31:13.231398 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.231351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzsfq\" (UniqueName: \"kubernetes.io/projected/a8d687b9-8e77-442e-8522-0532fd561fa0-kube-api-access-gzsfq\") pod \"a8d687b9-8e77-442e-8522-0532fd561fa0\" (UID: \"a8d687b9-8e77-442e-8522-0532fd561fa0\") "
Sep 17 23:31:13.231594 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.231570    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d687b9-8e77-442e-8522-0532fd561fa0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8d687b9-8e77-442e-8522-0532fd561fa0" (UID: "a8d687b9-8e77-442e-8522-0532fd561fa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:13.231670 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.231635    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqg29\" (UniqueName: \"kubernetes.io/projected/3973213c-acc3-4030-a510-19690f545bd6-kube-api-access-pqg29\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:13.233620 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.233587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d687b9-8e77-442e-8522-0532fd561fa0-kube-api-access-gzsfq" (OuterVolumeSpecName: "kube-api-access-gzsfq") pod "a8d687b9-8e77-442e-8522-0532fd561fa0" (UID: "a8d687b9-8e77-442e-8522-0532fd561fa0"). InnerVolumeSpecName "kube-api-access-gzsfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:13.333073 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.332984    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d687b9-8e77-442e-8522-0532fd561fa0-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:13.333073 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.333016    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzsfq\" (UniqueName: \"kubernetes.io/projected/a8d687b9-8e77-442e-8522-0532fd561fa0-kube-api-access-gzsfq\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:13.344251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.344226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:13.860957 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:13.860925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:14.043405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.043375    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"
Sep 17 23:31:14.043572 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.043396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb" event={"ID":"a8d687b9-8e77-442e-8522-0532fd561fa0","Type":"ContainerDied","Data":"f041c879fbd9dc1d62b9693487fe7f804f4273824e3b202c9b884101dfbf1a61"}
Sep 17 23:31:14.043572 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.043481    2580 scope.go:117] "RemoveContainer" containerID="fb1f60ef543b4efab67880ec7c8c3e040c7f1a2ff75b05b06ebdeaa6a85da713"
Sep 17 23:31:14.043698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.043614    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"
Sep 17 23:31:14.072527 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.072451    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"]
Sep 17 23:31:14.076584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.076557    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ab633-68894c99c5-c8n98"]
Sep 17 23:31:14.085912 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.085885    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"]
Sep 17 23:31:14.090296 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.090269    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7ab633-7c46f8dc6b-jgjpb"]
Sep 17 23:31:14.109788 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.109755    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3973213c-acc3-4030-a510-19690f545bd6" path="/var/lib/kubelet/pods/3973213c-acc3-4030-a510-19690f545bd6/volumes"
Sep 17 23:31:14.110197 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.110182    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a6a01d-ea3b-4b11-bd63-d61fcfa3849c" path="/var/lib/kubelet/pods/96a6a01d-ea3b-4b11-bd63-d61fcfa3849c/volumes"
Sep 17 23:31:14.110552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.110539    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d687b9-8e77-442e-8522-0532fd561fa0" path="/var/lib/kubelet/pods/a8d687b9-8e77-442e-8522-0532fd561fa0/volumes"
Sep 17 23:31:14.651250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:14.651213    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:15.334765 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:15.334725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:16.105078 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:16.105034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:16.789934 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:16.789897    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:16.890485 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:16.890450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:17.067697 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.067610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:17.381363 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381283    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"]
Sep 17 23:31:17.381668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381656    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a6a01d-ea3b-4b11-bd63-d61fcfa3849c" containerName="revoke-keys"
Sep 17 23:31:17.381714 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381670    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a6a01d-ea3b-4b11-bd63-d61fcfa3849c" containerName="revoke-keys"
Sep 17 23:31:17.381714 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381691    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d687b9-8e77-442e-8522-0532fd561fa0" containerName="payload-pre-processing"
Sep 17 23:31:17.381714 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381697    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d687b9-8e77-442e-8522-0532fd561fa0" containerName="payload-pre-processing"
Sep 17 23:31:17.381714 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381703    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3973213c-acc3-4030-a510-19690f545bd6" containerName="payload-processing"
Sep 17 23:31:17.381714 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381709    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3973213c-acc3-4030-a510-19690f545bd6" containerName="payload-processing"
Sep 17 23:31:17.381936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381717    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy"
Sep 17 23:31:17.381936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381723    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy"
Sep 17 23:31:17.381936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381779    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3973213c-acc3-4030-a510-19690f545bd6" containerName="payload-processing"
Sep 17 23:31:17.381936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381787    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c76884-ca63-40e3-b4d0-9a7f9a71d691" containerName="istio-proxy"
Sep 17 23:31:17.381936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381795    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a6a01d-ea3b-4b11-bd63-d61fcfa3849c" containerName="revoke-keys"
Sep 17 23:31:17.381936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.381818    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d687b9-8e77-442e-8522-0532fd561fa0" containerName="payload-pre-processing"
Sep 17 23:31:17.386024 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.386001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"
Sep 17 23:31:17.388741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.388717    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:31:17.389179 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.389162    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:31:17.393364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.393338    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"]
Sep 17 23:31:17.535646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.535610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:17.571650 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.571617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cecd91f-9fda-4b6f-975f-8b0e3ee63c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l\" (UID: \"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"
Sep 17 23:31:17.673081 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.673039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cecd91f-9fda-4b6f-975f-8b0e3ee63c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l\" (UID: \"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"
Sep 17 23:31:17.673680 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.673657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cecd91f-9fda-4b6f-975f-8b0e3ee63c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l\" (UID: \"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"
Sep 17 23:31:17.698971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.698938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"
Sep 17 23:31:17.971329 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:17.970365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"]
Sep 17 23:31:17.974168 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:31:17.974136    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cecd91f_9fda_4b6f_975f_8b0e3ee63c53.slice/crio-27e8e365a8e0432907a06f0bf2d010860a7af9331d81f6b91b5764a3ee142f1f WatchSource:0}: Error finding container 27e8e365a8e0432907a06f0bf2d010860a7af9331d81f6b91b5764a3ee142f1f: Status 404 returned error can't find the container with id 27e8e365a8e0432907a06f0bf2d010860a7af9331d81f6b91b5764a3ee142f1f
Sep 17 23:31:18.068285 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:18.068250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l" event={"ID":"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53","Type":"ContainerStarted","Data":"27e8e365a8e0432907a06f0bf2d010860a7af9331d81f6b91b5764a3ee142f1f"}
Sep 17 23:31:18.420503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:18.420467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:19.074752 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:19.074659    2580 generic.go:356] "Generic (PLEG): container finished" podID="2cecd91f-9fda-4b6f-975f-8b0e3ee63c53" containerID="7925be1152308af565631b880e641f8f220d5915e413fa1bfb3a74470bf692c0" exitCode=0
Sep 17 23:31:19.074752 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:19.074740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l" event={"ID":"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53","Type":"ContainerDied","Data":"7925be1152308af565631b880e641f8f220d5915e413fa1bfb3a74470bf692c0"}
Sep 17 23:31:19.143203 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:19.143169    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:19.945006 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:19.944974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:20.099269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.099229    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr" podUID="1ebdb1e5-d445-4cda-bfd0-54b8138a8363" containerName="main" containerID="cri-o://3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086" gracePeriod=45
Sep 17 23:31:20.104136 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.103995    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q" podUID="1809a149-bd3c-4497-84e3-308e9015968b" containerName="main" containerID="cri-o://990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9" gracePeriod=45
Sep 17 23:31:20.120627 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.120600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:20.124067 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.124022    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz" podUID="2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" containerName="main" containerID="cri-o://e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2" gracePeriod=45
Sep 17 23:31:20.223121 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.223047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:20.295471 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.295058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"
Sep 17 23:31:20.370845 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.370468    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:31:20.384492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.384410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:20.400288 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.400264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cecd91f-9fda-4b6f-975f-8b0e3ee63c53-maas-api-service-ca\") pod \"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53\" (UID: \"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53\") "
Sep 17 23:31:20.400696 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.400666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cecd91f-9fda-4b6f-975f-8b0e3ee63c53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2cecd91f-9fda-4b6f-975f-8b0e3ee63c53" (UID: "2cecd91f-9fda-4b6f-975f-8b0e3ee63c53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:20.415411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.415380    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:31:20.458901 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.458867    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:31:20.501186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tmp-dir\") pod \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") "
Sep 17 23:31:20.501186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501161    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tls-certs\") pod \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") "
Sep 17 23:31:20.501186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501185    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-dshm\") pod \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") "
Sep 17 23:31:20.501461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-home\") pod \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") "
Sep 17 23:31:20.501461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-model-cache\") pod \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") "
Sep 17 23:31:20.501461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ebdb1e5-d445-4cda-bfd0-54b8138a8363" (UID: "1ebdb1e5-d445-4cda-bfd0-54b8138a8363"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.501461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwwjd\" (UniqueName: \"kubernetes.io/projected/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-kube-api-access-nwwjd\") pod \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\" (UID: \"1ebdb1e5-d445-4cda-bfd0-54b8138a8363\") "
Sep 17 23:31:20.502007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501727    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cecd91f-9fda-4b6f-975f-8b0e3ee63c53-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.502007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501752    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.502007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-home" (OuterVolumeSpecName: "home") pod "1ebdb1e5-d445-4cda-bfd0-54b8138a8363" (UID: "1ebdb1e5-d445-4cda-bfd0-54b8138a8363"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.502007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.501985    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ebdb1e5-d445-4cda-bfd0-54b8138a8363" (UID: "1ebdb1e5-d445-4cda-bfd0-54b8138a8363"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.503864 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.503835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ebdb1e5-d445-4cda-bfd0-54b8138a8363" (UID: "1ebdb1e5-d445-4cda-bfd0-54b8138a8363"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:20.503991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.503974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-kube-api-access-nwwjd" (OuterVolumeSpecName: "kube-api-access-nwwjd") pod "1ebdb1e5-d445-4cda-bfd0-54b8138a8363" (UID: "1ebdb1e5-d445-4cda-bfd0-54b8138a8363"). InnerVolumeSpecName "kube-api-access-nwwjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:20.504053 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.503996    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-dshm" (OuterVolumeSpecName: "dshm") pod "1ebdb1e5-d445-4cda-bfd0-54b8138a8363" (UID: "1ebdb1e5-d445-4cda-bfd0-54b8138a8363"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.602092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-home\") pod \"1809a149-bd3c-4497-84e3-308e9015968b\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") "
Sep 17 23:31:20.602290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-tmp-dir\") pod \"1809a149-bd3c-4497-84e3-308e9015968b\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") "
Sep 17 23:31:20.602290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6wx5\" (UniqueName: \"kubernetes.io/projected/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-kube-api-access-k6wx5\") pod \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") "
Sep 17 23:31:20.602290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602177    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-model-cache\") pod \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") "
Sep 17 23:31:20.602290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-dshm\") pod \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") "
Sep 17 23:31:20.602290 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-997wh\" (UniqueName: \"kubernetes.io/projected/1809a149-bd3c-4497-84e3-308e9015968b-kube-api-access-997wh\") pod \"1809a149-bd3c-4497-84e3-308e9015968b\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") "
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tls-certs\") pod \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") "
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-home\") pod \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") "
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-dshm\") pod \"1809a149-bd3c-4497-84e3-308e9015968b\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") "
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-model-cache\") pod \"1809a149-bd3c-4497-84e3-308e9015968b\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") "
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-home" (OuterVolumeSpecName: "home") pod "1809a149-bd3c-4497-84e3-308e9015968b" (UID: "1809a149-bd3c-4497-84e3-308e9015968b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1809a149-bd3c-4497-84e3-308e9015968b" (UID: "1809a149-bd3c-4497-84e3-308e9015968b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tmp-dir\") pod \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\" (UID: \"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf\") "
Sep 17 23:31:20.602566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602487    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1809a149-bd3c-4497-84e3-308e9015968b-tls-certs\") pod \"1809a149-bd3c-4497-84e3-308e9015968b\" (UID: \"1809a149-bd3c-4497-84e3-308e9015968b\") "
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-home" (OuterVolumeSpecName: "home") pod "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" (UID: "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" (UID: "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602855    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602875    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwjd\" (UniqueName: \"kubernetes.io/projected/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-kube-api-access-nwwjd\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602890    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602902    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602917    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602929    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebdb1e5-d445-4cda-bfd0-54b8138a8363-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602940    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.602951    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.603008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.603001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1809a149-bd3c-4497-84e3-308e9015968b" (UID: "1809a149-bd3c-4497-84e3-308e9015968b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.603717 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.603134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" (UID: "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.605345 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.605295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" (UID: "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:20.605914 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.605889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-dshm" (OuterVolumeSpecName: "dshm") pod "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" (UID: "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.606313 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.606284    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1809a149-bd3c-4497-84e3-308e9015968b-kube-api-access-997wh" (OuterVolumeSpecName: "kube-api-access-997wh") pod "1809a149-bd3c-4497-84e3-308e9015968b" (UID: "1809a149-bd3c-4497-84e3-308e9015968b"). InnerVolumeSpecName "kube-api-access-997wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:20.606866 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.606840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-kube-api-access-k6wx5" (OuterVolumeSpecName: "kube-api-access-k6wx5") pod "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" (UID: "2c8df38b-006a-4b5d-9b52-2a5a66d2cabf"). InnerVolumeSpecName "kube-api-access-k6wx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:20.607461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.607436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-dshm" (OuterVolumeSpecName: "dshm") pod "1809a149-bd3c-4497-84e3-308e9015968b" (UID: "1809a149-bd3c-4497-84e3-308e9015968b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:20.607591 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.607573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1809a149-bd3c-4497-84e3-308e9015968b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1809a149-bd3c-4497-84e3-308e9015968b" (UID: "1809a149-bd3c-4497-84e3-308e9015968b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:20.703539 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703500    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703539 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703536    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703551    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1809a149-bd3c-4497-84e3-308e9015968b-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703563    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703575    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1809a149-bd3c-4497-84e3-308e9015968b-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703587    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6wx5\" (UniqueName: \"kubernetes.io/projected/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-kube-api-access-k6wx5\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703599    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703613    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.703741 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.703625    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997wh\" (UniqueName: \"kubernetes.io/projected/1809a149-bd3c-4497-84e3-308e9015968b-kube-api-access-997wh\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:20.844575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:20.844483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:21.089533 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.089497    2580 generic.go:356] "Generic (PLEG): container finished" podID="1809a149-bd3c-4497-84e3-308e9015968b" containerID="990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9" exitCode=0
Sep 17 23:31:21.089734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.089563    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"
Sep 17 23:31:21.089734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.089598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q" event={"ID":"1809a149-bd3c-4497-84e3-308e9015968b","Type":"ContainerDied","Data":"990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9"}
Sep 17 23:31:21.089734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.089656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q" event={"ID":"1809a149-bd3c-4497-84e3-308e9015968b","Type":"ContainerDied","Data":"50b30e6772bddd2fa00cfabb86f80292d0dd243e022d60b937e166c8e23d61ee"}
Sep 17 23:31:21.089734 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.089682    2580 scope.go:117] "RemoveContainer" containerID="990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9"
Sep 17 23:31:21.091188 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.091167    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" containerID="e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2" exitCode=0
Sep 17 23:31:21.091258 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.091235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"
Sep 17 23:31:21.091307 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.091265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz" event={"ID":"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf","Type":"ContainerDied","Data":"e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2"}
Sep 17 23:31:21.091307 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.091296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz" event={"ID":"2c8df38b-006a-4b5d-9b52-2a5a66d2cabf","Type":"ContainerDied","Data":"41b53b5f44138c18902503c1ed48c570dd4ae80ed1f2f8889102f914aa049058"}
Sep 17 23:31:21.093040 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.093018    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"
Sep 17 23:31:21.093159 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.093023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l" event={"ID":"2cecd91f-9fda-4b6f-975f-8b0e3ee63c53","Type":"ContainerDied","Data":"27e8e365a8e0432907a06f0bf2d010860a7af9331d81f6b91b5764a3ee142f1f"}
Sep 17 23:31:21.093223 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.093182    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e8e365a8e0432907a06f0bf2d010860a7af9331d81f6b91b5764a3ee142f1f"
Sep 17 23:31:21.095646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.095586    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ebdb1e5-d445-4cda-bfd0-54b8138a8363" containerID="3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086" exitCode=0
Sep 17 23:31:21.095646 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.095635    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"
Sep 17 23:31:21.095820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.095632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr" event={"ID":"1ebdb1e5-d445-4cda-bfd0-54b8138a8363","Type":"ContainerDied","Data":"3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086"}
Sep 17 23:31:21.095820 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.095716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr" event={"ID":"1ebdb1e5-d445-4cda-bfd0-54b8138a8363","Type":"ContainerDied","Data":"ebd301d4613a8fc38d8ec9054a29d3bb9b959eb1cfb71abb08c15b00a5ea3185"}
Sep 17 23:31:21.100246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.100223    2580 scope.go:117] "RemoveContainer" containerID="990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9"
Sep 17 23:31:21.100595 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:31:21.100563    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9\": container with ID starting with 990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9 not found: ID does not exist" containerID="990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9"
Sep 17 23:31:21.100657 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.100585    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9"} err="failed to get container status \"990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9\": rpc error: code = NotFound desc = could not find container \"990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9\": container with ID starting with 990ad42ac8c13a850dcf17583b31901d99f3fa0dd25fa4555b27b9adec2424b9 not found: ID does not exist"
Sep 17 23:31:21.100657 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.100608    2580 scope.go:117] "RemoveContainer" containerID="e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2"
Sep 17 23:31:21.110615 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.110596    2580 scope.go:117] "RemoveContainer" containerID="e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2"
Sep 17 23:31:21.110921 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:31:21.110902    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2\": container with ID starting with e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2 not found: ID does not exist" containerID="e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2"
Sep 17 23:31:21.111009 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.110932    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2"} err="failed to get container status \"e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2\": rpc error: code = NotFound desc = could not find container \"e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2\": container with ID starting with e670cbb6b766d80b4fc3dcd570db3e877dd5b6a125b3cce86a45b33b85c735d2 not found: ID does not exist"
Sep 17 23:31:21.111009 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.110953    2580 scope.go:117] "RemoveContainer" containerID="3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086"
Sep 17 23:31:21.120997 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.120978    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"]
Sep 17 23:31:21.126238 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.126218    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-w4-7ab633-kserve-74cbf9bf45-n7h6q"]
Sep 17 23:31:21.132974 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.132954    2580 scope.go:117] "RemoveContainer" containerID="3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086"
Sep 17 23:31:21.133297 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:31:21.133277    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086\": container with ID starting with 3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086 not found: ID does not exist" containerID="3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086"
Sep 17 23:31:21.133365 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.133302    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086"} err="failed to get container status \"3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086\": rpc error: code = NotFound desc = could not find container \"3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086\": container with ID starting with 3122e81b8e47feb31a873a4545812f783cc3f77285321bcf797140baa7392086 not found: ID does not exist"
Sep 17 23:31:21.141414 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.141386    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"]
Sep 17 23:31:21.146037 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.146019    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/simulator-w4-7ab633-kserve-6cb7bc9955-s7qkz"]
Sep 17 23:31:21.158728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.158702    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"]
Sep 17 23:31:21.162929 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:21.162907    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ab633/distinct-2-w4-7ab633-kserve-6fd845b8d6-899hr"]
Sep 17 23:31:22.107764 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:22.107728    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1809a149-bd3c-4497-84e3-308e9015968b" path="/var/lib/kubelet/pods/1809a149-bd3c-4497-84e3-308e9015968b/volumes"
Sep 17 23:31:22.108199 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:22.108184    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebdb1e5-d445-4cda-bfd0-54b8138a8363" path="/var/lib/kubelet/pods/1ebdb1e5-d445-4cda-bfd0-54b8138a8363/volumes"
Sep 17 23:31:22.108553 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:22.108540    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" path="/var/lib/kubelet/pods/2c8df38b-006a-4b5d-9b52-2a5a66d2cabf/volumes"
Sep 17 23:31:27.854281 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:27.854244    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"]
Sep 17 23:31:27.859263 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:27.859237    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-55130c-29828130-b9djp"]
Sep 17 23:31:27.859879 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:27.859848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:27.885926 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:27.885890    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"]
Sep 17 23:31:27.886971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:27.886464    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7" podUID="8bfbface-810f-497c-a320-63774cf51403" containerName="payload-processing" containerID="cri-o://da134918811af9b92a8848ae6a5ce2287596ccc5f9649c24eca03f9822f80423" gracePeriod=30
Sep 17 23:31:27.959514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:27.959484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:28.053455 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.053423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"]
Sep 17 23:31:28.056994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.056940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:28.057134 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.057083    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-55130c-7zz9l"]
Sep 17 23:31:28.111339 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.111250    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1004eb63-7572-4d03-94fb-0fb092855ab5" path="/var/lib/kubelet/pods/1004eb63-7572-4d03-94fb-0fb092855ab5/volumes"
Sep 17 23:31:28.111568 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.111554    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cecd91f-9fda-4b6f-975f-8b0e3ee63c53" path="/var/lib/kubelet/pods/2cecd91f-9fda-4b6f-975f-8b0e3ee63c53/volumes"
Sep 17 23:31:28.128598 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.128568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:28.140480 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.140453    2580 generic.go:356] "Generic (PLEG): container finished" podID="8bfbface-810f-497c-a320-63774cf51403" containerID="da134918811af9b92a8848ae6a5ce2287596ccc5f9649c24eca03f9822f80423" exitCode=0
Sep 17 23:31:28.140585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.140520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7" event={"ID":"8bfbface-810f-497c-a320-63774cf51403","Type":"ContainerDied","Data":"da134918811af9b92a8848ae6a5ce2287596ccc5f9649c24eca03f9822f80423"}
Sep 17 23:31:28.140585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.140566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7" event={"ID":"8bfbface-810f-497c-a320-63774cf51403","Type":"ContainerDied","Data":"02f043ad5159bce788d866814d8fecceadc2a874b41fd8f240a2b590270c29ab"}
Sep 17 23:31:28.140585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.140583    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f043ad5159bce788d866814d8fecceadc2a874b41fd8f240a2b590270c29ab"
Sep 17 23:31:28.152853 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.152835    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:31:28.165321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.165300    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-httwc\" (UniqueName: \"kubernetes.io/projected/8bfbface-810f-497c-a320-63774cf51403-kube-api-access-httwc\") pod \"8bfbface-810f-497c-a320-63774cf51403\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") "
Sep 17 23:31:28.165418 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.165346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfbface-810f-497c-a320-63774cf51403-plugins-config-volume\") pod \"8bfbface-810f-497c-a320-63774cf51403\" (UID: \"8bfbface-810f-497c-a320-63774cf51403\") "
Sep 17 23:31:28.165762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.165732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfbface-810f-497c-a320-63774cf51403-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bfbface-810f-497c-a320-63774cf51403" (UID: "8bfbface-810f-497c-a320-63774cf51403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:28.167858 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.167833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfbface-810f-497c-a320-63774cf51403-kube-api-access-httwc" (OuterVolumeSpecName: "kube-api-access-httwc") pod "8bfbface-810f-497c-a320-63774cf51403" (UID: "8bfbface-810f-497c-a320-63774cf51403"). InnerVolumeSpecName "kube-api-access-httwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:28.266193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.266154    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-httwc\" (UniqueName: \"kubernetes.io/projected/8bfbface-810f-497c-a320-63774cf51403-kube-api-access-httwc\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:28.266193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.266185    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfbface-810f-497c-a320-63774cf51403-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:31:28.407155 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.407113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:28.511484 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:28.511443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:29.146220 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:29.146186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"
Sep 17 23:31:29.170582 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:29.170556    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"]
Sep 17 23:31:29.177114 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:29.177088    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-55130c-6b6fcb6dfc-wmtl7"]
Sep 17 23:31:29.356011 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:29.355977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:30.043762 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:30.043721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:30.110133 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:30.110086    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfbface-810f-497c-a320-63774cf51403" path="/var/lib/kubelet/pods/8bfbface-810f-497c-a320-63774cf51403/volumes"
Sep 17 23:31:30.139227 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:30.139194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:31:30.139417 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:30.139269    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:31:30.260214 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:30.260182    2580 scope.go:117] "RemoveContainer" containerID="da134918811af9b92a8848ae6a5ce2287596ccc5f9649c24eca03f9822f80423"
Sep 17 23:31:30.269185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:30.269164    2580 scope.go:117] "RemoveContainer" containerID="29bcadf7d2cbf70c41304de5a45d4344b1665fd7b9412bc926bfb4c9fdd2df2c"
Sep 17 23:31:34.548036 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:34.547999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:34.618886 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:34.618828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:34.824087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:34.824009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:42.118701 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:42.118665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:42.218826 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:42.218766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:42.499071 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:42.499027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:52.989410 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:31:52.989322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:04.071210 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:04.071172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:05.552737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:05.552701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:08.007772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:08.007731    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:08.823317 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:08.823283    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"]
Sep 17 23:32:08.843092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:08.843055    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:17.846375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.846340    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"]
Sep 17 23:32:17.846959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.846929    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cecd91f-9fda-4b6f-975f-8b0e3ee63c53" containerName="revoke-keys"
Sep 17 23:32:17.846959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.846956    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cecd91f-9fda-4b6f-975f-8b0e3ee63c53" containerName="revoke-keys"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.846973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebdb1e5-d445-4cda-bfd0-54b8138a8363" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.846990    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebdb1e5-d445-4cda-bfd0-54b8138a8363" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847006    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1809a149-bd3c-4497-84e3-308e9015968b" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1809a149-bd3c-4497-84e3-308e9015968b" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847055    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847073    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfbface-810f-497c-a320-63774cf51403" containerName="payload-processing"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847081    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfbface-810f-497c-a320-63774cf51403" containerName="payload-processing"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847175    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8df38b-006a-4b5d-9b52-2a5a66d2cabf" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847189    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1809a149-bd3c-4497-84e3-308e9015968b" containerName="main"
Sep 17 23:32:17.847190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847196    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cecd91f-9fda-4b6f-975f-8b0e3ee63c53" containerName="revoke-keys"
Sep 17 23:32:17.847768 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847203    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfbface-810f-497c-a320-63774cf51403" containerName="payload-processing"
Sep 17 23:32:17.847768 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.847211    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebdb1e5-d445-4cda-bfd0-54b8138a8363" containerName="main"
Sep 17 23:32:17.850432 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.850412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:17.852870 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.852849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5aae7\"/\"sub-test-model-e5aae7-kserve-self-signed-certs\""
Sep 17 23:32:17.852870 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.852857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5aae7\"/\"kube-root-ca.crt\""
Sep 17 23:32:17.854044 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.854024    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5aae7\"/\"default-dockercfg-plbq2\""
Sep 17 23:32:17.854128 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.854053    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5aae7\"/\"openshift-service-ca.crt\""
Sep 17 23:32:17.858726 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.858703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"]
Sep 17 23:32:17.901002 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.900973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:17.927595 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.927566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-dshm\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:17.927763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.927611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwc5l\" (UniqueName: \"kubernetes.io/projected/09051f58-b23c-415b-8ebc-3683486ff3a1-kube-api-access-cwc5l\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:17.927763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.927649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-home\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:17.927763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.927666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-tmp-dir\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:17.927763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.927690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09051f58-b23c-415b-8ebc-3683486ff3a1-tls-certs\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:17.927763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:17.927727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-model-cache\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-model-cache\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-dshm\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwc5l\" (UniqueName: \"kubernetes.io/projected/09051f58-b23c-415b-8ebc-3683486ff3a1-kube-api-access-cwc5l\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-home\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-tmp-dir\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09051f58-b23c-415b-8ebc-3683486ff3a1-tls-certs\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-model-cache\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-home\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.029769 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.029661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-tmp-dir\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.031724 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.031701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-dshm\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.031907 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.031891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09051f58-b23c-415b-8ebc-3683486ff3a1-tls-certs\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.038351 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.038326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwc5l\" (UniqueName: \"kubernetes.io/projected/09051f58-b23c-415b-8ebc-3683486ff3a1-kube-api-access-cwc5l\") pod \"sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.161091 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.161057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:18.434599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:18.434572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"]
Sep 17 23:32:18.437003 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:32:18.436968    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09051f58_b23c_415b_8ebc_3683486ff3a1.slice/crio-845a01910693ebce6f853f7c005ac83e3b1d105eb63c59c5c38e04eae69dbe7f WatchSource:0}: Error finding container 845a01910693ebce6f853f7c005ac83e3b1d105eb63c59c5c38e04eae69dbe7f: Status 404 returned error can't find the container with id 845a01910693ebce6f853f7c005ac83e3b1d105eb63c59c5c38e04eae69dbe7f
Sep 17 23:32:19.411941 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:19.411904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s" event={"ID":"09051f58-b23c-415b-8ebc-3683486ff3a1","Type":"ContainerStarted","Data":"1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9"}
Sep 17 23:32:19.411941 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:19.411945    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s" event={"ID":"09051f58-b23c-415b-8ebc-3683486ff3a1","Type":"ContainerStarted","Data":"845a01910693ebce6f853f7c005ac83e3b1d105eb63c59c5c38e04eae69dbe7f"}
Sep 17 23:32:19.431212 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:19.431164    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s" podStartSLOduration=2.431148105 podStartE2EDuration="2.431148105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:19.429286774 +0000 UTC m=+1549.859146593" watchObservedRunningTime="2026-09-17 23:32:19.431148105 +0000 UTC m=+1549.861007923"
Sep 17 23:32:23.843079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:23.843010    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc" podUID="0fdff86f-117e-40dc-bf45-0b56bc2654b7" containerName="main" containerID="cri-o://2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d" gracePeriod=45
Sep 17 23:32:24.099757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.099692    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:32:24.193183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tmp-dir\") pod \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") "
Sep 17 23:32:24.193371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-dshm\") pod \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") "
Sep 17 23:32:24.193371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tls-certs\") pod \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") "
Sep 17 23:32:24.193371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-home\") pod \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") "
Sep 17 23:32:24.193371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-model-cache\") pod \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") "
Sep 17 23:32:24.193565 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt47h\" (UniqueName: \"kubernetes.io/projected/0fdff86f-117e-40dc-bf45-0b56bc2654b7-kube-api-access-qt47h\") pod \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\" (UID: \"0fdff86f-117e-40dc-bf45-0b56bc2654b7\") "
Sep 17 23:32:24.193565 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fdff86f-117e-40dc-bf45-0b56bc2654b7" (UID: "0fdff86f-117e-40dc-bf45-0b56bc2654b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:24.193668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-home" (OuterVolumeSpecName: "home") pod "0fdff86f-117e-40dc-bf45-0b56bc2654b7" (UID: "0fdff86f-117e-40dc-bf45-0b56bc2654b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:24.193907 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fdff86f-117e-40dc-bf45-0b56bc2654b7" (UID: "0fdff86f-117e-40dc-bf45-0b56bc2654b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:24.193907 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193886    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.193907 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.193926    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.195777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.195748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-dshm" (OuterVolumeSpecName: "dshm") pod "0fdff86f-117e-40dc-bf45-0b56bc2654b7" (UID: "0fdff86f-117e-40dc-bf45-0b56bc2654b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:24.195777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.195761    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdff86f-117e-40dc-bf45-0b56bc2654b7-kube-api-access-qt47h" (OuterVolumeSpecName: "kube-api-access-qt47h") pod "0fdff86f-117e-40dc-bf45-0b56bc2654b7" (UID: "0fdff86f-117e-40dc-bf45-0b56bc2654b7"). InnerVolumeSpecName "kube-api-access-qt47h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:24.195777 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.195748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fdff86f-117e-40dc-bf45-0b56bc2654b7" (UID: "0fdff86f-117e-40dc-bf45-0b56bc2654b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:24.295424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.295381    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt47h\" (UniqueName: \"kubernetes.io/projected/0fdff86f-117e-40dc-bf45-0b56bc2654b7-kube-api-access-qt47h\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.295424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.295412    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.295424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.295422    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fdff86f-117e-40dc-bf45-0b56bc2654b7-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.295424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.295431    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fdff86f-117e-40dc-bf45-0b56bc2654b7-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.442106 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.442069    2580 generic.go:356] "Generic (PLEG): container finished" podID="0fdff86f-117e-40dc-bf45-0b56bc2654b7" containerID="2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d" exitCode=0
Sep 17 23:32:24.442293 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.442113    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"
Sep 17 23:32:24.442293 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.442162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc" event={"ID":"0fdff86f-117e-40dc-bf45-0b56bc2654b7","Type":"ContainerDied","Data":"2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d"}
Sep 17 23:32:24.442293 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.442205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc" event={"ID":"0fdff86f-117e-40dc-bf45-0b56bc2654b7","Type":"ContainerDied","Data":"7c6997b8c4505c045e4f53be5b86576e30dcc54ff2daffed703f09e7d0782638"}
Sep 17 23:32:24.442293 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.442226    2580 scope.go:117] "RemoveContainer" containerID="2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d"
Sep 17 23:32:24.451623 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.451605    2580 scope.go:117] "RemoveContainer" containerID="2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d"
Sep 17 23:32:24.451898 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:32:24.451876    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d\": container with ID starting with 2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d not found: ID does not exist" containerID="2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d"
Sep 17 23:32:24.452008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.451904    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d"} err="failed to get container status \"2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d\": rpc error: code = NotFound desc = could not find container \"2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d\": container with ID starting with 2587083b5f1226a6af60ba7a00e47ad0782da903b5335a885da3cdeee6bab48d not found: ID does not exist"
Sep 17 23:32:24.463678 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.463642    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"]
Sep 17 23:32:24.467230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:24.467211    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/auth-test-model-6275e9-kserve-795cdfbb99-mnqdc"]
Sep 17 23:32:26.108408 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:26.108370    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fdff86f-117e-40dc-bf45-0b56bc2654b7" path="/var/lib/kubelet/pods/0fdff86f-117e-40dc-bf45-0b56bc2654b7/volumes"
Sep 17 23:32:28.162198 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:28.162161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:28.162584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:28.162243    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:28.175227 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:28.175202    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:28.474272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:28.474190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:32:43.957398 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.957366    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"]
Sep 17 23:32:43.958060 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.957983    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fdff86f-117e-40dc-bf45-0b56bc2654b7" containerName="main"
Sep 17 23:32:43.958060 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.958002    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fdff86f-117e-40dc-bf45-0b56bc2654b7" containerName="main"
Sep 17 23:32:43.958181 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.958085    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fdff86f-117e-40dc-bf45-0b56bc2654b7" containerName="main"
Sep 17 23:32:43.962350 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.962327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:43.965716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.965694    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"kube-root-ca.crt\""
Sep 17 23:32:43.965954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.965693    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"default-dockercfg-vtrc6\""
Sep 17 23:32:43.965954 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.965752    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"openshift-service-ca.crt\""
Sep 17 23:32:43.966153 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.965752    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"sub-test-model-6275e9-kserve-self-signed-certs\""
Sep 17 23:32:43.979619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.979583    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"]
Sep 17 23:32:43.993750 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:43.993711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:44.085144 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.085106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-tmp-dir\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.085321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.085184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d899d5c-9566-4485-87f9-6e69a5e1c757-tls-certs\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.085321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.085211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-dshm\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.085321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.085257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96fr\" (UniqueName: \"kubernetes.io/projected/2d899d5c-9566-4485-87f9-6e69a5e1c757-kube-api-access-f96fr\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.085321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.085289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-model-cache\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.085496 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.085432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-home\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186304 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-home\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186304 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-tmp-dir\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d899d5c-9566-4485-87f9-6e69a5e1c757-tls-certs\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-dshm\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96fr\" (UniqueName: \"kubernetes.io/projected/2d899d5c-9566-4485-87f9-6e69a5e1c757-kube-api-access-f96fr\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-model-cache\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186748 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-home\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.186922 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-model-cache\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.187012 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.186888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-tmp-dir\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.189311 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.189289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-dshm\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.189475 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.189458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d899d5c-9566-4485-87f9-6e69a5e1c757-tls-certs\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.194651 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.194628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96fr\" (UniqueName: \"kubernetes.io/projected/2d899d5c-9566-4485-87f9-6e69a5e1c757-kube-api-access-f96fr\") pod \"sub-test-model-6275e9-kserve-8dfd4fc95-pqs66\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.273644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.273556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:44.548684 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:44.548602    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"]
Sep 17 23:32:44.552073 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:32:44.552038    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d899d5c_9566_4485_87f9_6e69a5e1c757.slice/crio-12f2f54f2fd8c050f56f00c76d30eba87feddb18de38df0b6262026f172b2491 WatchSource:0}: Error finding container 12f2f54f2fd8c050f56f00c76d30eba87feddb18de38df0b6262026f172b2491: Status 404 returned error can't find the container with id 12f2f54f2fd8c050f56f00c76d30eba87feddb18de38df0b6262026f172b2491
Sep 17 23:32:45.554427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:45.554387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66" event={"ID":"2d899d5c-9566-4485-87f9-6e69a5e1c757","Type":"ContainerStarted","Data":"b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168"}
Sep 17 23:32:45.554427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:45.554430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66" event={"ID":"2d899d5c-9566-4485-87f9-6e69a5e1c757","Type":"ContainerStarted","Data":"12f2f54f2fd8c050f56f00c76d30eba87feddb18de38df0b6262026f172b2491"}
Sep 17 23:32:45.574421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:45.574366    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66" podStartSLOduration=2.574349464 podStartE2EDuration="2.574349464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:45.573842734 +0000 UTC m=+1576.003702552" watchObservedRunningTime="2026-09-17 23:32:45.574349464 +0000 UTC m=+1576.004209282"
Sep 17 23:32:54.274784 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:54.274742    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:54.274784 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:54.274794    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:54.287703 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:54.287679    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:32:54.616990 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:32:54.616908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:33:28.049134 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:28.049054    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"]
Sep 17 23:33:28.071178 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:28.071134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:28.962524 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:28.962493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"]
Sep 17 23:33:28.979988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:28.979954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:30.329948 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:30.329916    2580 scope.go:117] "RemoveContainer" containerID="fd168e7fa321dad6dc3e715b6608319c77d7f11b90cbfd34b08d73fc9baa1f12"
Sep 17 23:33:41.440274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.440235    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"]
Sep 17 23:33:41.452628 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.452595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.455492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.455454    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e5aae7\"/\"rate-test-model-e5aae7-kserve-self-signed-certs\""
Sep 17 23:33:41.458262 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.458235    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"]
Sep 17 23:33:41.482486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.482454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:41.534947 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.534914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-model-cache\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.535116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.534974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tmp-dir\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.535116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.535017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tls-certs\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.535116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.535061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-home\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.535116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.535090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvv2x\" (UniqueName: \"kubernetes.io/projected/2ac28013-383d-41c6-b0e6-ab9a324ccb47-kube-api-access-xvv2x\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.535116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.535113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-dshm\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.635874 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.635836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tmp-dir\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.635874 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.635886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tls-certs\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.636125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.635922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-home\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.636125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.635943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvv2x\" (UniqueName: \"kubernetes.io/projected/2ac28013-383d-41c6-b0e6-ab9a324ccb47-kube-api-access-xvv2x\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.636125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.635966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-dshm\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.636125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.636107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-model-cache\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.636345 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.636235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tmp-dir\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.636345 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.636264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-home\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.636459 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.636442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-model-cache\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.638382 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.638356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-dshm\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.638614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.638596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tls-certs\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.644169 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.644145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvv2x\" (UniqueName: \"kubernetes.io/projected/2ac28013-383d-41c6-b0e6-ab9a324ccb47-kube-api-access-xvv2x\") pod \"rate-test-model-e5aae7-kserve-57686db59-486cd\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") " pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:41.765111 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:41.765028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:42.040272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:42.040247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"]
Sep 17 23:33:42.042776 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:33:42.042748    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ac28013_383d_41c6_b0e6_ab9a324ccb47.slice/crio-a828b917a5ddacfeb22655606c2046ac111a69387719a2c82fc793d848475563 WatchSource:0}: Error finding container a828b917a5ddacfeb22655606c2046ac111a69387719a2c82fc793d848475563: Status 404 returned error can't find the container with id a828b917a5ddacfeb22655606c2046ac111a69387719a2c82fc793d848475563
Sep 17 23:33:42.867379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:42.867341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd" event={"ID":"2ac28013-383d-41c6-b0e6-ab9a324ccb47","Type":"ContainerStarted","Data":"d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5"}
Sep 17 23:33:42.867877 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:42.867391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd" event={"ID":"2ac28013-383d-41c6-b0e6-ab9a324ccb47","Type":"ContainerStarted","Data":"a828b917a5ddacfeb22655606c2046ac111a69387719a2c82fc793d848475563"}
Sep 17 23:33:42.887401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:42.887350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd" podStartSLOduration=1.8873245239999998 podStartE2EDuration="1.887324524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:42.885299191 +0000 UTC m=+1633.315159008" watchObservedRunningTime="2026-09-17 23:33:42.887324524 +0000 UTC m=+1633.317184341"
Sep 17 23:33:43.068919 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.068866    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s" podUID="09051f58-b23c-415b-8ebc-3683486ff3a1" containerName="main" containerID="cri-o://1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9" gracePeriod=45
Sep 17 23:33:43.314400 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.314372    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:33:43.453870 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.453765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-tmp-dir\") pod \"09051f58-b23c-415b-8ebc-3683486ff3a1\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") "
Sep 17 23:33:43.453870 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.453826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-home\") pod \"09051f58-b23c-415b-8ebc-3683486ff3a1\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") "
Sep 17 23:33:43.453870 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.453858    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-dshm\") pod \"09051f58-b23c-415b-8ebc-3683486ff3a1\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") "
Sep 17 23:33:43.454139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.453892    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-model-cache\") pod \"09051f58-b23c-415b-8ebc-3683486ff3a1\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") "
Sep 17 23:33:43.454139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.453917    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09051f58-b23c-415b-8ebc-3683486ff3a1-tls-certs\") pod \"09051f58-b23c-415b-8ebc-3683486ff3a1\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") "
Sep 17 23:33:43.454139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.453949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwc5l\" (UniqueName: \"kubernetes.io/projected/09051f58-b23c-415b-8ebc-3683486ff3a1-kube-api-access-cwc5l\") pod \"09051f58-b23c-415b-8ebc-3683486ff3a1\" (UID: \"09051f58-b23c-415b-8ebc-3683486ff3a1\") "
Sep 17 23:33:43.454139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.454075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09051f58-b23c-415b-8ebc-3683486ff3a1" (UID: "09051f58-b23c-415b-8ebc-3683486ff3a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:43.454364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.454147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "09051f58-b23c-415b-8ebc-3683486ff3a1" (UID: "09051f58-b23c-415b-8ebc-3683486ff3a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:43.454364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.454165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-home" (OuterVolumeSpecName: "home") pod "09051f58-b23c-415b-8ebc-3683486ff3a1" (UID: "09051f58-b23c-415b-8ebc-3683486ff3a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:43.454364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.454268    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:43.454364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.454286    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:43.454364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.454296    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:43.456297 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.456277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-dshm" (OuterVolumeSpecName: "dshm") pod "09051f58-b23c-415b-8ebc-3683486ff3a1" (UID: "09051f58-b23c-415b-8ebc-3683486ff3a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:43.456388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.456291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09051f58-b23c-415b-8ebc-3683486ff3a1-kube-api-access-cwc5l" (OuterVolumeSpecName: "kube-api-access-cwc5l") pod "09051f58-b23c-415b-8ebc-3683486ff3a1" (UID: "09051f58-b23c-415b-8ebc-3683486ff3a1"). InnerVolumeSpecName "kube-api-access-cwc5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:43.456388 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.456337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09051f58-b23c-415b-8ebc-3683486ff3a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09051f58-b23c-415b-8ebc-3683486ff3a1" (UID: "09051f58-b23c-415b-8ebc-3683486ff3a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:43.555016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.554980    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09051f58-b23c-415b-8ebc-3683486ff3a1-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:43.555016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.555011    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09051f58-b23c-415b-8ebc-3683486ff3a1-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:43.555016 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.555020    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwc5l\" (UniqueName: \"kubernetes.io/projected/09051f58-b23c-415b-8ebc-3683486ff3a1-kube-api-access-cwc5l\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:43.874517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.874421    2580 generic.go:356] "Generic (PLEG): container finished" podID="09051f58-b23c-415b-8ebc-3683486ff3a1" containerID="1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9" exitCode=0
Sep 17 23:33:43.874517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.874472    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"
Sep 17 23:33:43.874983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.874514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s" event={"ID":"09051f58-b23c-415b-8ebc-3683486ff3a1","Type":"ContainerDied","Data":"1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9"}
Sep 17 23:33:43.874983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.874560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s" event={"ID":"09051f58-b23c-415b-8ebc-3683486ff3a1","Type":"ContainerDied","Data":"845a01910693ebce6f853f7c005ac83e3b1d105eb63c59c5c38e04eae69dbe7f"}
Sep 17 23:33:43.874983 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.874577    2580 scope.go:117] "RemoveContainer" containerID="1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9"
Sep 17 23:33:43.884407 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.884382    2580 scope.go:117] "RemoveContainer" containerID="1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9"
Sep 17 23:33:43.884684 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:33:43.884665    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9\": container with ID starting with 1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9 not found: ID does not exist" containerID="1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9"
Sep 17 23:33:43.884749 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.884693    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9"} err="failed to get container status \"1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9\": rpc error: code = NotFound desc = could not find container \"1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9\": container with ID starting with 1cef787c6dbcbc69af6b98861da81b0551ff4633fcfaa2872355d4e65de900d9 not found: ID does not exist"
Sep 17 23:33:43.894702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.894670    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"]
Sep 17 23:33:43.898359 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.898334    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/sub-test-model-e5aae7-kserve-dcb877ff5-8hs5s"]
Sep 17 23:33:43.982754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:43.982714    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66" podUID="2d899d5c-9566-4485-87f9-6e69a5e1c757" containerName="main" containerID="cri-o://b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168" gracePeriod=45
Sep 17 23:33:44.109949 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.109912    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09051f58-b23c-415b-8ebc-3683486ff3a1" path="/var/lib/kubelet/pods/09051f58-b23c-415b-8ebc-3683486ff3a1/volumes"
Sep 17 23:33:44.245298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.245272    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:33:44.362529 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.362495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d899d5c-9566-4485-87f9-6e69a5e1c757-tls-certs\") pod \"2d899d5c-9566-4485-87f9-6e69a5e1c757\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") "
Sep 17 23:33:44.362716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.362598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96fr\" (UniqueName: \"kubernetes.io/projected/2d899d5c-9566-4485-87f9-6e69a5e1c757-kube-api-access-f96fr\") pod \"2d899d5c-9566-4485-87f9-6e69a5e1c757\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") "
Sep 17 23:33:44.362716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.362636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-dshm\") pod \"2d899d5c-9566-4485-87f9-6e69a5e1c757\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") "
Sep 17 23:33:44.362716 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.362707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-home\") pod \"2d899d5c-9566-4485-87f9-6e69a5e1c757\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") "
Sep 17 23:33:44.363019 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.362761    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-model-cache\") pod \"2d899d5c-9566-4485-87f9-6e69a5e1c757\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") "
Sep 17 23:33:44.363019 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.362794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-tmp-dir\") pod \"2d899d5c-9566-4485-87f9-6e69a5e1c757\" (UID: \"2d899d5c-9566-4485-87f9-6e69a5e1c757\") "
Sep 17 23:33:44.363137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.363057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d899d5c-9566-4485-87f9-6e69a5e1c757" (UID: "2d899d5c-9566-4485-87f9-6e69a5e1c757"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:44.363137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.363065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-home" (OuterVolumeSpecName: "home") pod "2d899d5c-9566-4485-87f9-6e69a5e1c757" (UID: "2d899d5c-9566-4485-87f9-6e69a5e1c757"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:44.363250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.363200    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d899d5c-9566-4485-87f9-6e69a5e1c757" (UID: "2d899d5c-9566-4485-87f9-6e69a5e1c757"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:44.363250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.363219    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:44.363504 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.363237    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:44.365075 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.365043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-dshm" (OuterVolumeSpecName: "dshm") pod "2d899d5c-9566-4485-87f9-6e69a5e1c757" (UID: "2d899d5c-9566-4485-87f9-6e69a5e1c757"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:44.365075 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.365056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d899d5c-9566-4485-87f9-6e69a5e1c757-kube-api-access-f96fr" (OuterVolumeSpecName: "kube-api-access-f96fr") pod "2d899d5c-9566-4485-87f9-6e69a5e1c757" (UID: "2d899d5c-9566-4485-87f9-6e69a5e1c757"). InnerVolumeSpecName "kube-api-access-f96fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:44.365224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.365084    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d899d5c-9566-4485-87f9-6e69a5e1c757-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d899d5c-9566-4485-87f9-6e69a5e1c757" (UID: "2d899d5c-9566-4485-87f9-6e69a5e1c757"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:44.464192 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.464106    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:44.464192 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.464135    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d899d5c-9566-4485-87f9-6e69a5e1c757-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:44.464192 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.464144    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96fr\" (UniqueName: \"kubernetes.io/projected/2d899d5c-9566-4485-87f9-6e69a5e1c757-kube-api-access-f96fr\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:44.464192 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.464155    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d899d5c-9566-4485-87f9-6e69a5e1c757-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:33:44.884059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.883958    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d899d5c-9566-4485-87f9-6e69a5e1c757" containerID="b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168" exitCode=0
Sep 17 23:33:44.884059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.884004    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"
Sep 17 23:33:44.884587 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.884005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66" event={"ID":"2d899d5c-9566-4485-87f9-6e69a5e1c757","Type":"ContainerDied","Data":"b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168"}
Sep 17 23:33:44.884587 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.884127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66" event={"ID":"2d899d5c-9566-4485-87f9-6e69a5e1c757","Type":"ContainerDied","Data":"12f2f54f2fd8c050f56f00c76d30eba87feddb18de38df0b6262026f172b2491"}
Sep 17 23:33:44.884587 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.884158    2580 scope.go:117] "RemoveContainer" containerID="b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168"
Sep 17 23:33:44.894936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.894913    2580 scope.go:117] "RemoveContainer" containerID="b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168"
Sep 17 23:33:44.895204 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:33:44.895184    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168\": container with ID starting with b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168 not found: ID does not exist" containerID="b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168"
Sep 17 23:33:44.895284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.895209    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168"} err="failed to get container status \"b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168\": rpc error: code = NotFound desc = could not find container \"b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168\": container with ID starting with b67f6d1190297ea5c2a07ed35f19d638a51aa936d124cea42c81f0ecd6573168 not found: ID does not exist"
Sep 17 23:33:44.904364 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.904342    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"]
Sep 17 23:33:44.907952 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:44.907928    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/sub-test-model-6275e9-kserve-8dfd4fc95-pqs66"]
Sep 17 23:33:46.108307 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:46.108265    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d899d5c-9566-4485-87f9-6e69a5e1c757" path="/var/lib/kubelet/pods/2d899d5c-9566-4485-87f9-6e69a5e1c757/volumes"
Sep 17 23:33:51.765879 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:51.765834    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:51.766356 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:51.765903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:51.778701 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:51.778673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:33:51.935395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:33:51.935367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:34:00.732603 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.732570    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"]
Sep 17 23:34:00.733004 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.732960    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d899d5c-9566-4485-87f9-6e69a5e1c757" containerName="main"
Sep 17 23:34:00.733004 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.732972    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d899d5c-9566-4485-87f9-6e69a5e1c757" containerName="main"
Sep 17 23:34:00.733004 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.732982    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09051f58-b23c-415b-8ebc-3683486ff3a1" containerName="main"
Sep 17 23:34:00.733004 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.732988    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09051f58-b23c-415b-8ebc-3683486ff3a1" containerName="main"
Sep 17 23:34:00.733130 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.733040    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09051f58-b23c-415b-8ebc-3683486ff3a1" containerName="main"
Sep 17 23:34:00.733130 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.733049    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d899d5c-9566-4485-87f9-6e69a5e1c757" containerName="main"
Sep 17 23:34:00.736043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.736020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:00.738377 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.738355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"default-dockercfg-vtrc6\""
Sep 17 23:34:00.739251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.739228    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"kube-root-ca.crt\""
Sep 17 23:34:00.739354 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.739232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"openshift-service-ca.crt\""
Sep 17 23:34:00.739354 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.739272    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6275e9\"/\"rate-test-model-6275e9-kserve-self-signed-certs\""
Sep 17 23:34:00.753768 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.753724    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"]
Sep 17 23:34:00.767024 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.766988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:00.924743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.924704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tls-certs\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:00.924743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.924746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-home\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:00.925041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.924768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlkq\" (UniqueName: \"kubernetes.io/projected/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-kube-api-access-tvlkq\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:00.925041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.924944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-dshm\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:00.925041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.925033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tmp-dir\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:00.925185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:00.925057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-model-cache\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026300 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-dshm\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tmp-dir\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-model-cache\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tls-certs\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-home\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlkq\" (UniqueName: \"kubernetes.io/projected/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-kube-api-access-tvlkq\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tmp-dir\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.026885 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-model-cache\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.027006 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.026921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-home\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.028686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.028663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-dshm\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.029001 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.028984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tls-certs\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.034357 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.034338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlkq\" (UniqueName: \"kubernetes.io/projected/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-kube-api-access-tvlkq\") pod \"rate-test-model-6275e9-kserve-545499c95-dphtg\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") " pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.048287 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.048262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:01.323681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.323652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"]
Sep 17 23:34:01.325672 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:34:01.325627    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d4166da_d3b1_4e8c_8a11_d5f9c94d4190.slice/crio-80d31a38e8a8117d9843977d94d9d3ca73ca165df04fad9db900f7778bd9846c WatchSource:0}: Error finding container 80d31a38e8a8117d9843977d94d9d3ca73ca165df04fad9db900f7778bd9846c: Status 404 returned error can't find the container with id 80d31a38e8a8117d9843977d94d9d3ca73ca165df04fad9db900f7778bd9846c
Sep 17 23:34:01.976772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.976733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg" event={"ID":"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190","Type":"ContainerStarted","Data":"283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87"}
Sep 17 23:34:01.977183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.976782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg" event={"ID":"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190","Type":"ContainerStarted","Data":"80d31a38e8a8117d9843977d94d9d3ca73ca165df04fad9db900f7778bd9846c"}
Sep 17 23:34:01.992998 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:01.992938    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg" podStartSLOduration=1.9929022440000002 podStartE2EDuration="1.992902244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:01.991746155 +0000 UTC m=+1652.421605975" watchObservedRunningTime="2026-09-17 23:34:01.992902244 +0000 UTC m=+1652.422762065"
Sep 17 23:34:07.590833 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:07.590763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:07.656385 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:07.656345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:11.048917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:11.048885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:11.049316 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:11.048935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:11.061408 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:11.061377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:12.042896 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:12.042861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:40.435230 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:40.435192    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"]
Sep 17 23:34:40.453178 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:40.453073    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:41.235789 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:41.235750    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"]
Sep 17 23:34:41.255457 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:41.255419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:41.679513 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:41.679480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:41.982622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:41.982531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:42.232471 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:42.232436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:44.560182 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:44.560145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:44.798833 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:44.798780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:44.900554 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:44.900527    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:45.557593 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:45.557559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:46.399688 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:46.399651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:46.500062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:46.500019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:46.600106 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:46.600065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:46.865894 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:46.865792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:46.955860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:46.955826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:47.021510 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.021477    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"]
Sep 17 23:34:47.025232 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.025215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.028166 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.028134    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-93a496\""
Sep 17 23:34:47.028166 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.028151    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-93a496-dockercfg-9s4tc\""
Sep 17 23:34:47.040609 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.040581    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"]
Sep 17 23:34:47.061448 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.061420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2px\" (UniqueName: \"kubernetes.io/projected/9bc0f952-4e2b-43df-95af-8eaab94aeec7-kube-api-access-mb2px\") pod \"payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.061610 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.061461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc0f952-4e2b-43df-95af-8eaab94aeec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.162074 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.162039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2px\" (UniqueName: \"kubernetes.io/projected/9bc0f952-4e2b-43df-95af-8eaab94aeec7-kube-api-access-mb2px\") pod \"payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.162276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.162093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc0f952-4e2b-43df-95af-8eaab94aeec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.162750 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.162729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc0f952-4e2b-43df-95af-8eaab94aeec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.174573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.174542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2px\" (UniqueName: \"kubernetes.io/projected/9bc0f952-4e2b-43df-95af-8eaab94aeec7-kube-api-access-mb2px\") pod \"payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.335997 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.335957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:47.616660 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:47.616635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"]
Sep 17 23:34:47.618852 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:34:47.618822    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc0f952_4e2b_43df_95af_8eaab94aeec7.slice/crio-8455f326b1255c854a614545c79a8c99f13996813e1b675d7b3b9dc1367aff31 WatchSource:0}: Error finding container 8455f326b1255c854a614545c79a8c99f13996813e1b675d7b3b9dc1367aff31: Status 404 returned error can't find the container with id 8455f326b1255c854a614545c79a8c99f13996813e1b675d7b3b9dc1367aff31
Sep 17 23:34:48.228763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:48.228725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q" event={"ID":"9bc0f952-4e2b-43df-95af-8eaab94aeec7","Type":"ContainerStarted","Data":"c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a"}
Sep 17 23:34:48.228984 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:48.228779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q" event={"ID":"9bc0f952-4e2b-43df-95af-8eaab94aeec7","Type":"ContainerStarted","Data":"8455f326b1255c854a614545c79a8c99f13996813e1b675d7b3b9dc1367aff31"}
Sep 17 23:34:48.228984 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:48.228917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:34:48.245636 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:48.245586    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q" podStartSLOduration=2.245570024 podStartE2EDuration="2.245570024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:48.244641694 +0000 UTC m=+1698.674501512" watchObservedRunningTime="2026-09-17 23:34:48.245570024 +0000 UTC m=+1698.675429842"
Sep 17 23:34:55.455523 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.455424    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd" podUID="2ac28013-383d-41c6-b0e6-ab9a324ccb47" containerName="main" containerID="cri-o://d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5" gracePeriod=45
Sep 17 23:34:55.710608 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.710544    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:34:55.743515 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.743486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tls-certs\") pod \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") "
Sep 17 23:34:55.743664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.743560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvv2x\" (UniqueName: \"kubernetes.io/projected/2ac28013-383d-41c6-b0e6-ab9a324ccb47-kube-api-access-xvv2x\") pod \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") "
Sep 17 23:34:55.743664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.743594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tmp-dir\") pod \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") "
Sep 17 23:34:55.743664 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.743636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-dshm\") pod \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") "
Sep 17 23:34:55.743852 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.743664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-model-cache\") pod \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") "
Sep 17 23:34:55.743852 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.743692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-home\") pod \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\" (UID: \"2ac28013-383d-41c6-b0e6-ab9a324ccb47\") "
Sep 17 23:34:55.744004 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.743972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ac28013-383d-41c6-b0e6-ab9a324ccb47" (UID: "2ac28013-383d-41c6-b0e6-ab9a324ccb47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:55.744092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.744070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ac28013-383d-41c6-b0e6-ab9a324ccb47" (UID: "2ac28013-383d-41c6-b0e6-ab9a324ccb47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:55.744237 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.744211    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-home" (OuterVolumeSpecName: "home") pod "2ac28013-383d-41c6-b0e6-ab9a324ccb47" (UID: "2ac28013-383d-41c6-b0e6-ab9a324ccb47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:55.746105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.746069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ac28013-383d-41c6-b0e6-ab9a324ccb47" (UID: "2ac28013-383d-41c6-b0e6-ab9a324ccb47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:34:55.746105 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.746079    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac28013-383d-41c6-b0e6-ab9a324ccb47-kube-api-access-xvv2x" (OuterVolumeSpecName: "kube-api-access-xvv2x") pod "2ac28013-383d-41c6-b0e6-ab9a324ccb47" (UID: "2ac28013-383d-41c6-b0e6-ab9a324ccb47"). InnerVolumeSpecName "kube-api-access-xvv2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:34:55.746367 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.746347    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-dshm" (OuterVolumeSpecName: "dshm") pod "2ac28013-383d-41c6-b0e6-ab9a324ccb47" (UID: "2ac28013-383d-41c6-b0e6-ab9a324ccb47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:55.845056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.845019    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:55.845056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.845051    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvv2x\" (UniqueName: \"kubernetes.io/projected/2ac28013-383d-41c6-b0e6-ab9a324ccb47-kube-api-access-xvv2x\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:55.845056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.845062    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:55.845303 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.845074    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:55.845303 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.845082    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:55.845303 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:55.845090    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac28013-383d-41c6-b0e6-ab9a324ccb47-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:56.255348 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.255307    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg" podUID="9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" containerName="main" containerID="cri-o://283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87" gracePeriod=45
Sep 17 23:34:56.273000 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.272966    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac28013-383d-41c6-b0e6-ab9a324ccb47" containerID="d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5" exitCode=0
Sep 17 23:34:56.273119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.273026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"
Sep 17 23:34:56.273119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.273045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd" event={"ID":"2ac28013-383d-41c6-b0e6-ab9a324ccb47","Type":"ContainerDied","Data":"d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5"}
Sep 17 23:34:56.273119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.273097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd" event={"ID":"2ac28013-383d-41c6-b0e6-ab9a324ccb47","Type":"ContainerDied","Data":"a828b917a5ddacfeb22655606c2046ac111a69387719a2c82fc793d848475563"}
Sep 17 23:34:56.273119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.273115    2580 scope.go:117] "RemoveContainer" containerID="d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5"
Sep 17 23:34:56.328405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.328366    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"]
Sep 17 23:34:56.332452 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.332429    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e5aae7/rate-test-model-e5aae7-kserve-57686db59-486cd"]
Sep 17 23:34:56.366786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.366765    2580 scope.go:117] "RemoveContainer" containerID="d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5"
Sep 17 23:34:56.367172 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:34:56.367146    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5\": container with ID starting with d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5 not found: ID does not exist" containerID="d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5"
Sep 17 23:34:56.367241 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.367181    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5"} err="failed to get container status \"d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5\": rpc error: code = NotFound desc = could not find container \"d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5\": container with ID starting with d33cf0ff2b99ac3e8d6d4f9aa30e9a0330b41a7ef15c5b8480063e38ff7054a5 not found: ID does not exist"
Sep 17 23:34:56.506516 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.506492    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:56.550433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550400    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvlkq\" (UniqueName: \"kubernetes.io/projected/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-kube-api-access-tvlkq\") pod \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") "
Sep 17 23:34:56.550433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tmp-dir\") pod \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") "
Sep 17 23:34:56.550695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tls-certs\") pod \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") "
Sep 17 23:34:56.550695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-dshm\") pod \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") "
Sep 17 23:34:56.550695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-home\") pod \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") "
Sep 17 23:34:56.550695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550655    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-model-cache\") pod \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\" (UID: \"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190\") "
Sep 17 23:34:56.550968 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" (UID: "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:56.551031 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550966    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-home" (OuterVolumeSpecName: "home") pod "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" (UID: "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:56.551031 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.550976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" (UID: "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:56.551140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.551054    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:56.551140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.551072    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:56.551140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.551086    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:56.552831 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.552792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-dshm" (OuterVolumeSpecName: "dshm") pod "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" (UID: "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:56.552831 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.552818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-kube-api-access-tvlkq" (OuterVolumeSpecName: "kube-api-access-tvlkq") pod "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" (UID: "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190"). InnerVolumeSpecName "kube-api-access-tvlkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:34:56.552989 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.552848    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" (UID: "9d4166da-d3b1-4e8c-8a11-d5f9c94d4190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:34:56.652312 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.652281    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlkq\" (UniqueName: \"kubernetes.io/projected/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-kube-api-access-tvlkq\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:56.652312 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.652312    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:56.652496 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:56.652322    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:34:57.280026 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.279989    2580 generic.go:356] "Generic (PLEG): container finished" podID="9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" containerID="283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87" exitCode=0
Sep 17 23:34:57.280026 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.280028    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"
Sep 17 23:34:57.280259 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.280035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg" event={"ID":"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190","Type":"ContainerDied","Data":"283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87"}
Sep 17 23:34:57.280259 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.280070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg" event={"ID":"9d4166da-d3b1-4e8c-8a11-d5f9c94d4190","Type":"ContainerDied","Data":"80d31a38e8a8117d9843977d94d9d3ca73ca165df04fad9db900f7778bd9846c"}
Sep 17 23:34:57.280259 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.280087    2580 scope.go:117] "RemoveContainer" containerID="283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87"
Sep 17 23:34:57.290872 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.290846    2580 scope.go:117] "RemoveContainer" containerID="283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87"
Sep 17 23:34:57.291166 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:34:57.291145    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87\": container with ID starting with 283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87 not found: ID does not exist" containerID="283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87"
Sep 17 23:34:57.291242 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.291174    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87"} err="failed to get container status \"283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87\": rpc error: code = NotFound desc = could not find container \"283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87\": container with ID starting with 283bfc9350072cf0bc1a7f3dea017d9d62c3f6a2076ce47f55577dcf1c12de87 not found: ID does not exist"
Sep 17 23:34:57.300811 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.300769    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"]
Sep 17 23:34:57.305514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:57.305491    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6275e9/rate-test-model-6275e9-kserve-545499c95-dphtg"]
Sep 17 23:34:58.110236 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:58.110204    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac28013-383d-41c6-b0e6-ab9a324ccb47" path="/var/lib/kubelet/pods/2ac28013-383d-41c6-b0e6-ab9a324ccb47/volumes"
Sep 17 23:34:58.110617 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:58.110597    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" path="/var/lib/kubelet/pods/9d4166da-d3b1-4e8c-8a11-d5f9c94d4190/volumes"
Sep 17 23:34:59.236466 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:34:59.236441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:35:08.298955 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:08.298913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:09.027339 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:09.027303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:09.709962 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:09.709931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:09.809752 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:09.809721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:10.372275 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.372245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:10.469266 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.469234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:10.650644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.650611    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"]
Sep 17 23:35:10.651062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.651041    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac28013-383d-41c6-b0e6-ab9a324ccb47" containerName="main"
Sep 17 23:35:10.651062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.651058    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac28013-383d-41c6-b0e6-ab9a324ccb47" containerName="main"
Sep 17 23:35:10.651248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.651071    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" containerName="main"
Sep 17 23:35:10.651248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.651078    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" containerName="main"
Sep 17 23:35:10.651248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.651134    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac28013-383d-41c6-b0e6-ab9a324ccb47" containerName="main"
Sep 17 23:35:10.651248 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.651144    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4166da-d3b1-4e8c-8a11-d5f9c94d4190" containerName="main"
Sep 17 23:35:10.654126 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.654109    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.657162 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.657142    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86dda1-openshift-default-dockercfg-54h2v\""
Sep 17 23:35:10.666687 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.666664    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"]
Sep 17 23:35:10.681494 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.681468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:10.785193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-socket\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-data\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-envoy\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6frg\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-kube-api-access-q6frg\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-credential-socket\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-certs\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-token\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-podinfo\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.785566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.785483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756fd94-7c01-40b2-96f3-098a0cce75dd-istiod-ca-cert\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886541 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-credential-socket\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886693 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-certs\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886693 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-token\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886693 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-podinfo\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886693 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756fd94-7c01-40b2-96f3-098a0cce75dd-istiod-ca-cert\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-socket\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-data\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-envoy\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.886959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.886874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6frg\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-kube-api-access-q6frg\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.887175 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.887012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-credential-socket\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.887235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.887166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-data\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.887235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.887207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-socket\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.887407 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.887361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-certs\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.887567 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.887544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756fd94-7c01-40b2-96f3-098a0cce75dd-istiod-ca-cert\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.889305 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.889282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-envoy\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.889532 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.889508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-podinfo\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.894662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.894637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-token\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.895537 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.895515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6frg\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-kube-api-access-q6frg\") pod \"e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") " pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:10.965170 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:10.965072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:11.008622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.008589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:11.245421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.245394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"]
Sep 17 23:35:11.247726 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:35:11.247695    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4756fd94_7c01_40b2_96f3_098a0cce75dd.slice/crio-24e869de348586b2b4d6871de51e4e540011ec4e9d8e1ad13c8ef3dc7aeb8e40 WatchSource:0}: Error finding container 24e869de348586b2b4d6871de51e4e540011ec4e9d8e1ad13c8ef3dc7aeb8e40: Status 404 returned error can't find the container with id 24e869de348586b2b4d6871de51e4e540011ec4e9d8e1ad13c8ef3dc7aeb8e40
Sep 17 23:35:11.250080 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.250046    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:35:11.250171 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.250111    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:35:11.250171 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.250140    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:35:11.360209 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.360175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" event={"ID":"4756fd94-7c01-40b2-96f3-098a0cce75dd","Type":"ContainerStarted","Data":"e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73"}
Sep 17 23:35:11.360416 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.360232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" event={"ID":"4756fd94-7c01-40b2-96f3-098a0cce75dd","Type":"ContainerStarted","Data":"24e869de348586b2b4d6871de51e4e540011ec4e9d8e1ad13c8ef3dc7aeb8e40"}
Sep 17 23:35:11.380668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.380613    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podStartSLOduration=1.380595546 podStartE2EDuration="1.380595546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:11.378622049 +0000 UTC m=+1721.808481868" watchObservedRunningTime="2026-09-17 23:35:11.380595546 +0000 UTC m=+1721.810455364"
Sep 17 23:35:11.965905 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:11.965867    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:12.529563 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:12.529518    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:12.971334 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:12.971303    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:13.204881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:13.204850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:13.307623 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:13.307514    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:13.372109 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:13.372070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:13.373056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:13.373029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:13.401969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:13.401935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:14.160150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:14.160106    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:14.830383 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:14.830350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:15.709270 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:15.709235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:15.810571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:15.810532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:15.910303 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:15.910269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:16.164295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.164270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:16.263427 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.263396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:16.266422 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.266393    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"]
Sep 17 23:35:16.270169 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.270153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.272731 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.272713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86dda1-dockercfg-p86j4\""
Sep 17 23:35:16.272887 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.272758    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86dda1\""
Sep 17 23:35:16.286792 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.286765    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"]
Sep 17 23:35:16.444488 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.444382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kkx\" (UniqueName: \"kubernetes.io/projected/a6e4706c-7b4e-4b6c-8478-daa1618b144d-kube-api-access-n4kkx\") pod \"payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.444488 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.444459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e4706c-7b4e-4b6c-8478-daa1618b144d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.545192 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.545156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kkx\" (UniqueName: \"kubernetes.io/projected/a6e4706c-7b4e-4b6c-8478-daa1618b144d-kube-api-access-n4kkx\") pod \"payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.545389 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.545208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e4706c-7b4e-4b6c-8478-daa1618b144d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.545898 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.545880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e4706c-7b4e-4b6c-8478-daa1618b144d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.553362 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.553335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kkx\" (UniqueName: \"kubernetes.io/projected/a6e4706c-7b4e-4b6c-8478-daa1618b144d-kube-api-access-n4kkx\") pod \"payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.580311 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.580274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:16.856402 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:16.856375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"]
Sep 17 23:35:16.859228 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:35:16.859198    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e4706c_7b4e_4b6c_8478_daa1618b144d.slice/crio-27b83271960e53c7b54a912d050f9876e64616870a074785022ba60d5c2dc833 WatchSource:0}: Error finding container 27b83271960e53c7b54a912d050f9876e64616870a074785022ba60d5c2dc833: Status 404 returned error can't find the container with id 27b83271960e53c7b54a912d050f9876e64616870a074785022ba60d5c2dc833
Sep 17 23:35:17.396577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:17.396532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6" event={"ID":"a6e4706c-7b4e-4b6c-8478-daa1618b144d","Type":"ContainerStarted","Data":"0be8e21bfec540b5c2d8dd722332ea38687aa0b27753dee48921147971b8830b"}
Sep 17 23:35:17.396577 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:17.396583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6" event={"ID":"a6e4706c-7b4e-4b6c-8478-daa1618b144d","Type":"ContainerStarted","Data":"27b83271960e53c7b54a912d050f9876e64616870a074785022ba60d5c2dc833"}
Sep 17 23:35:17.396881 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:17.396718    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:17.415102 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:17.415046    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6" podStartSLOduration=1.415024393 podStartE2EDuration="1.415024393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:17.413461114 +0000 UTC m=+1727.843320932" watchObservedRunningTime="2026-09-17 23:35:17.415024393 +0000 UTC m=+1727.844884212"
Sep 17 23:35:21.779370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:21.779334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:27.201842 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:27.201794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:28.404250 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:28.404218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:35:29.414023 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:29.413984    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 23:35:29.414419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:29.414050    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 23:35:30.473171 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:30.473138    2580 generic.go:356] "Generic (PLEG): container finished" podID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerID="e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73" exitCode=0
Sep 17 23:35:30.473583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:30.473191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" event={"ID":"4756fd94-7c01-40b2-96f3-098a0cce75dd","Type":"ContainerDied","Data":"e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73"}
Sep 17 23:35:30.473666 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:30.473647    2580 scope.go:117] "RemoveContainer" containerID="e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73"
Sep 17 23:35:30.474693 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:30.474662    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:35:30.474835 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:30.474754    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:35:30.474835 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:30.474825    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:35:31.480573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:31.480536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" event={"ID":"4756fd94-7c01-40b2-96f3-098a0cce75dd","Type":"ContainerStarted","Data":"03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02"}
Sep 17 23:35:31.966056 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:31.966020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:31.966264 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:31.966242    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 17 23:35:31.966342 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:31.966292    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 17 23:35:32.966425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:32.966386    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 17 23:35:32.966859 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:32.966450    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 17 23:35:33.277043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:33.276952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:33.966401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:33.966366    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 17 23:35:33.966584 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:33.966420    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 17 23:35:34.965770 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:34.965728    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 17 23:35:34.965976 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:34.965823    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 17 23:35:35.965945 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:35.965906    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 17 23:35:35.966340 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:35.965963    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 17 23:35:36.654376 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:36.654334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:36.969825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:36.969725    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:36.970256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:36.970154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:36.970982 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:36.970959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:35:37.867124 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:37.867089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:38.243503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:38.243467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:38.734745 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:38.734703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:39.329606 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:39.329568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:40.047310 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:40.047277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:40.723298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:40.723265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:54.540235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.540197    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"]
Sep 17 23:35:54.544152 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.544130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.546550 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.546527    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-93a496\"/\"openshift-service-ca.crt\""
Sep 17 23:35:54.547279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.547257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-93a496\"/\"default-dockercfg-x6rcb\""
Sep 17 23:35:54.547358 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.547277    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-93a496\"/\"kube-root-ca.crt\""
Sep 17 23:35:54.547358 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.547287    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-93a496\"/\"ipp-route-93a496-kserve-self-signed-certs\""
Sep 17 23:35:54.554472 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.554447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"]
Sep 17 23:35:54.568145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.568119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:54.724677 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.724646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-dshm\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.724893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.724710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whjl\" (UniqueName: \"kubernetes.io/projected/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-kube-api-access-6whjl\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.724893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.724737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tls-certs\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.724893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.724759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tmp-dir\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.724893 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.724782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-home\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.725100 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.724926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-model-cache\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-dshm\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whjl\" (UniqueName: \"kubernetes.io/projected/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-kube-api-access-6whjl\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tls-certs\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tmp-dir\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-home\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-model-cache\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826597 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tmp-dir\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826597 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-home\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.826989 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.826602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-model-cache\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.828572 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.828547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-dshm\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.828758 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.828744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tls-certs\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.833781 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.833762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whjl\" (UniqueName: \"kubernetes.io/projected/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-kube-api-access-6whjl\") pod \"ipp-route-93a496-kserve-f8cf79c9-tzqh4\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") " pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:54.853810 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:54.853762    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:35:55.134559 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:55.134534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"]
Sep 17 23:35:55.136316 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:35:55.136287    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod697cfe44_a8d6_46ce_80f3_cfbf0fa04086.slice/crio-1b242e8266784544d25f85e23dc4393bcb3f489ef9d940890599c9cd0f99a05e WatchSource:0}: Error finding container 1b242e8266784544d25f85e23dc4393bcb3f489ef9d940890599c9cd0f99a05e: Status 404 returned error can't find the container with id 1b242e8266784544d25f85e23dc4393bcb3f489ef9d940890599c9cd0f99a05e
Sep 17 23:35:55.622510 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:55.622472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4" event={"ID":"697cfe44-a8d6-46ce-80f3-cfbf0fa04086","Type":"ContainerStarted","Data":"78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9"}
Sep 17 23:35:55.622510 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:55.622514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4" event={"ID":"697cfe44-a8d6-46ce-80f3-cfbf0fa04086","Type":"ContainerStarted","Data":"1b242e8266784544d25f85e23dc4393bcb3f489ef9d940890599c9cd0f99a05e"}
Sep 17 23:35:55.638669 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:35:55.638612    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4" podStartSLOduration=1.6385937259999999 podStartE2EDuration="1.638593726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:55.638062584 +0000 UTC m=+1766.067922403" watchObservedRunningTime="2026-09-17 23:35:55.638593726 +0000 UTC m=+1766.068453555"
Sep 17 23:36:04.822432 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:04.822391    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:04.854379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:04.854342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:36:04.854566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:04.854394    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:36:04.867085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:04.867056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:36:05.686240 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:05.686208    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:36:20.008401 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.008365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:20.407539 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.407502    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"]
Sep 17 23:36:20.411190 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.411167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.414979 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.414950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-145e15-openshift-default-dockercfg-6j557\""
Sep 17 23:36:20.427033 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.427006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"]
Sep 17 23:36:20.437226 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.437197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:20.466681 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.466656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-credential-socket\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.466867 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.466695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-certs\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.466867 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.466726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istiod-ca-cert\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.466867 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.466772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-envoy\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.466867 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.466826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-podinfo\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.466867 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.466853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-data\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.467147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.466891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-token\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.467147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.467017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpsb\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-kube-api-access-rlpsb\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.467147 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.467088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-socket\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.568842 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.568781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-credential-socket\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.568875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-certs\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.568932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istiod-ca-cert\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.568967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-envoy\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.569014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-podinfo\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.569050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-data\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.569096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-token\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.569130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpsb\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-kube-api-access-rlpsb\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.569190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-socket\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.569644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.569619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-socket\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.570121 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.570095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-credential-socket\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.570370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.570350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-certs\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.571005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.570980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istiod-ca-cert\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.573834 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.573786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-envoy\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.574675 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.574647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-data\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.577027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.577002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-podinfo\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.583247 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.583219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-token\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.583633 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.583611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpsb\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-kube-api-access-rlpsb\") pod \"e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") " pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:20.722414 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:20.722308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:21.007555 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.007473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"]
Sep 17 23:36:21.010525 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:36:21.010499    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba32f8a2_fbc5_4914_a60c_f3ce755f1e93.slice/crio-5cd314d03cb89af757415e034e4c70d0b90e84ddabfed1053560a1f705eb905d WatchSource:0}: Error finding container 5cd314d03cb89af757415e034e4c70d0b90e84ddabfed1053560a1f705eb905d: Status 404 returned error can't find the container with id 5cd314d03cb89af757415e034e4c70d0b90e84ddabfed1053560a1f705eb905d
Sep 17 23:36:21.012836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.012780    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:36:21.012936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.012869    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:36:21.012936 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.012895    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:36:21.158525 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.158476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:21.764847 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.764743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" event={"ID":"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93","Type":"ContainerStarted","Data":"64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7"}
Sep 17 23:36:21.764847 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.764810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" event={"ID":"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93","Type":"ContainerStarted","Data":"5cd314d03cb89af757415e034e4c70d0b90e84ddabfed1053560a1f705eb905d"}
Sep 17 23:36:21.795588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:21.795538    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podStartSLOduration=1.795521325 podStartE2EDuration="1.795521325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:21.794034866 +0000 UTC m=+1792.223894881" watchObservedRunningTime="2026-09-17 23:36:21.795521325 +0000 UTC m=+1792.225381143"
Sep 17 23:36:22.723493 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:22.723454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:22.729185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:22.729157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:22.770705 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:22.770676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:22.771731 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:22.771711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:23.030790 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:23.030660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:23.683882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:23.683812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:23.778496 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:23.778450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:23.879971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:23.879926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:24.635212 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:24.635175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:25.320059 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:25.320023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:26.112815 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:26.112777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:26.954836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:26.954789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:27.057590 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.057559    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:27.155177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.155139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:27.433186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.433159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:27.513151 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.513117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:27.551853 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.551815    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"]
Sep 17 23:36:27.555639 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.555621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:27.558136 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.558111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-145e15\""
Sep 17 23:36:27.558489 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.558470    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-145e15-dockercfg-pdn7s\""
Sep 17 23:36:27.567013 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.566988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"]
Sep 17 23:36:27.643993 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.643952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vzq\" (UniqueName: \"kubernetes.io/projected/4dc45a81-340a-480c-9257-68ff117fea64-kube-api-access-s8vzq\") pod \"payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:27.644184 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.644022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc45a81-340a-480c-9257-68ff117fea64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:27.745498 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.745396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vzq\" (UniqueName: \"kubernetes.io/projected/4dc45a81-340a-480c-9257-68ff117fea64-kube-api-access-s8vzq\") pod \"payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:27.745498 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.745456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc45a81-340a-480c-9257-68ff117fea64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:27.746185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.746164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc45a81-340a-480c-9257-68ff117fea64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:27.753495 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.753477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vzq\" (UniqueName: \"kubernetes.io/projected/4dc45a81-340a-480c-9257-68ff117fea64-kube-api-access-s8vzq\") pod \"payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:27.867439 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:27.867404    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:28.146234 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:28.146057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"]
Sep 17 23:36:28.148710 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:36:28.148683    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc45a81_340a_480c_9257_68ff117fea64.slice/crio-84e85283a88fdfe08b5b2f8f08c13cc243b014edaf2f2c0ef292d82440b13fac WatchSource:0}: Error finding container 84e85283a88fdfe08b5b2f8f08c13cc243b014edaf2f2c0ef292d82440b13fac: Status 404 returned error can't find the container with id 84e85283a88fdfe08b5b2f8f08c13cc243b014edaf2f2c0ef292d82440b13fac
Sep 17 23:36:28.808827 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:28.808703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm" event={"ID":"4dc45a81-340a-480c-9257-68ff117fea64","Type":"ContainerStarted","Data":"04bb79f4f32ef3626bcd379217fcc650e259515e22d35905b49dc1f995d67ecd"}
Sep 17 23:36:28.808827 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:28.808768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm" event={"ID":"4dc45a81-340a-480c-9257-68ff117fea64","Type":"ContainerStarted","Data":"84e85283a88fdfe08b5b2f8f08c13cc243b014edaf2f2c0ef292d82440b13fac"}
Sep 17 23:36:28.809066 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:28.808908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:28.827181 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:28.827119    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm" podStartSLOduration=1.827080389 podStartE2EDuration="1.827080389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:28.824614353 +0000 UTC m=+1799.254474172" watchObservedRunningTime="2026-09-17 23:36:28.827080389 +0000 UTC m=+1799.256940208"
Sep 17 23:36:30.173843 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:30.173791    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:36:30.174573 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:30.174554    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:36:30.382474 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:30.382440    2580 scope.go:117] "RemoveContainer" containerID="d6d2cc4c0f982f3b79e440818a3654e8ce274700ee345cc42127610787111cb0"
Sep 17 23:36:30.391920 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:30.391887    2580 scope.go:117] "RemoveContainer" containerID="9781f82069cc7de23a912717a89a755e6659676375bbaf51a352cffcd8d07202"
Sep 17 23:36:30.400201 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:30.400178    2580 scope.go:117] "RemoveContainer" containerID="28f21ba693315aff421fc4521fafd5e9cd7ad37d1c247a92dee2cd4ed9a199f1"
Sep 17 23:36:34.454564 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:34.454534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:39.817872 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:39.817835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:40.750369 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:40.750330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:40.808403 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:40.808364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:42.886307 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:42.886274    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerID="64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7" exitCode=0
Sep 17 23:36:42.886730 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:42.886350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" event={"ID":"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93","Type":"ContainerDied","Data":"64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7"}
Sep 17 23:36:42.886912 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:42.886894    2580 scope.go:117] "RemoveContainer" containerID="64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7"
Sep 17 23:36:42.888212 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:42.888187    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:36:42.888285 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:42.888256    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:36:42.888328 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:42.888285    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 17 23:36:43.895038 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:43.894991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" event={"ID":"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93","Type":"ContainerStarted","Data":"dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397"}
Sep 17 23:36:44.722672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:44.722629    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:44.722964 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:44.722930    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 17 23:36:44.723027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:44.722993    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 17 23:36:45.722650 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:45.722619    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 17 23:36:45.723074 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:45.722676    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 17 23:36:46.723486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:46.723446    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 17 23:36:46.723915 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:46.723519    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 17 23:36:47.723156 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:47.723118    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 17 23:36:47.723340 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:47.723193    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 17 23:36:48.650787 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:48.650749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:48.723108 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:48.723074    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 17 23:36:48.723296 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:48.723145    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 17 23:36:48.908981 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:48.908880    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:49.399356 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.399316    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"]
Sep 17 23:36:49.403085 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.403061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"
Sep 17 23:36:49.409076 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.409042    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:36:49.409076 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.409065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:36:49.420017 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.419991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"]
Sep 17 23:36:49.462213 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.462155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f440566-7401-430c-bfa0-e8de0007431b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv\" (UID: \"3f440566-7401-430c-bfa0-e8de0007431b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"
Sep 17 23:36:49.563672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.563634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f440566-7401-430c-bfa0-e8de0007431b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv\" (UID: \"3f440566-7401-430c-bfa0-e8de0007431b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"
Sep 17 23:36:49.564287 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.564267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f440566-7401-430c-bfa0-e8de0007431b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv\" (UID: \"3f440566-7401-430c-bfa0-e8de0007431b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"
Sep 17 23:36:49.714120 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.714034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"
Sep 17 23:36:49.726852 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.726824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:49.727266 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.727241    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:49.727859 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.727840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:36:49.812702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.812660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:49.987727 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:49.987702    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"]
Sep 17 23:36:49.990002 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:36:49.989975    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f440566_7401_430c_bfa0_e8de0007431b.slice/crio-8e60956fd7e4f0983a4784be65e5b6179768afff2332792145d37bebf44db684 WatchSource:0}: Error finding container 8e60956fd7e4f0983a4784be65e5b6179768afff2332792145d37bebf44db684: Status 404 returned error can't find the container with id 8e60956fd7e4f0983a4784be65e5b6179768afff2332792145d37bebf44db684
Sep 17 23:36:50.745621 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:50.745585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:50.940150 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:50.940114    2580 generic.go:356] "Generic (PLEG): container finished" podID="3f440566-7401-430c-bfa0-e8de0007431b" containerID="734db7efc395a3f5ac485b18fce995d7938dc07ce7c8bf9e3b3ba6ca35412e3b" exitCode=0
Sep 17 23:36:50.940349 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:50.940197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv" event={"ID":"3f440566-7401-430c-bfa0-e8de0007431b","Type":"ContainerDied","Data":"734db7efc395a3f5ac485b18fce995d7938dc07ce7c8bf9e3b3ba6ca35412e3b"}
Sep 17 23:36:50.940349 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:50.940245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv" event={"ID":"3f440566-7401-430c-bfa0-e8de0007431b","Type":"ContainerStarted","Data":"8e60956fd7e4f0983a4784be65e5b6179768afff2332792145d37bebf44db684"}
Sep 17 23:36:51.441088 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:51.441059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:52.087292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.087261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:52.088426 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.088408    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"
Sep 17 23:36:52.192895 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.192856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f440566-7401-430c-bfa0-e8de0007431b-maas-api-service-ca\") pod \"3f440566-7401-430c-bfa0-e8de0007431b\" (UID: \"3f440566-7401-430c-bfa0-e8de0007431b\") "
Sep 17 23:36:52.193202 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.193180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f440566-7401-430c-bfa0-e8de0007431b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f440566-7401-430c-bfa0-e8de0007431b" (UID: "3f440566-7401-430c-bfa0-e8de0007431b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:36:52.293722 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.293635    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f440566-7401-430c-bfa0-e8de0007431b-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:36:52.845753 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.845675    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:52.952139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.952095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv" event={"ID":"3f440566-7401-430c-bfa0-e8de0007431b","Type":"ContainerDied","Data":"8e60956fd7e4f0983a4784be65e5b6179768afff2332792145d37bebf44db684"}
Sep 17 23:36:52.952309 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.952152    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e60956fd7e4f0983a4784be65e5b6179768afff2332792145d37bebf44db684"
Sep 17 23:36:52.952309 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:52.952115    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"
Sep 17 23:36:53.480281 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:53.480248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:55.087738 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.087707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:55.703139 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.703109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:55.730602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.730568    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"]
Sep 17 23:36:55.730916 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.730879    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm" podUID="4dc45a81-340a-480c-9257-68ff117fea64" containerName="payload-pre-processing" containerID="cri-o://04bb79f4f32ef3626bcd379217fcc650e259515e22d35905b49dc1f995d67ecd" gracePeriod=30
Sep 17 23:36:55.768076 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.768049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:55.868433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.868402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:55.889357 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.889324    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"]
Sep 17 23:36:55.895391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.895329    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145e15-msbbv"]
Sep 17 23:36:55.931476 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.931447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:55.970257 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.970228    2580 generic.go:356] "Generic (PLEG): container finished" podID="4dc45a81-340a-480c-9257-68ff117fea64" containerID="04bb79f4f32ef3626bcd379217fcc650e259515e22d35905b49dc1f995d67ecd" exitCode=0
Sep 17 23:36:55.970379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.970264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm" event={"ID":"4dc45a81-340a-480c-9257-68ff117fea64","Type":"ContainerDied","Data":"04bb79f4f32ef3626bcd379217fcc650e259515e22d35905b49dc1f995d67ecd"}
Sep 17 23:36:55.970379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.970299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm" event={"ID":"4dc45a81-340a-480c-9257-68ff117fea64","Type":"ContainerDied","Data":"84e85283a88fdfe08b5b2f8f08c13cc243b014edaf2f2c0ef292d82440b13fac"}
Sep 17 23:36:55.970379 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.970316    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84e85283a88fdfe08b5b2f8f08c13cc243b014edaf2f2c0ef292d82440b13fac"
Sep 17 23:36:55.981969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:55.981949    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:56.032238 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.032205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc45a81-340a-480c-9257-68ff117fea64-plugins-config-volume\") pod \"4dc45a81-340a-480c-9257-68ff117fea64\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") "
Sep 17 23:36:56.032425 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.032254    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8vzq\" (UniqueName: \"kubernetes.io/projected/4dc45a81-340a-480c-9257-68ff117fea64-kube-api-access-s8vzq\") pod \"4dc45a81-340a-480c-9257-68ff117fea64\" (UID: \"4dc45a81-340a-480c-9257-68ff117fea64\") "
Sep 17 23:36:56.032644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.032619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc45a81-340a-480c-9257-68ff117fea64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dc45a81-340a-480c-9257-68ff117fea64" (UID: "4dc45a81-340a-480c-9257-68ff117fea64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:36:56.034566 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.034546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc45a81-340a-480c-9257-68ff117fea64-kube-api-access-s8vzq" (OuterVolumeSpecName: "kube-api-access-s8vzq") pod "4dc45a81-340a-480c-9257-68ff117fea64" (UID: "4dc45a81-340a-480c-9257-68ff117fea64"). InnerVolumeSpecName "kube-api-access-s8vzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:36:56.109151 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.109113    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f440566-7401-430c-bfa0-e8de0007431b" path="/var/lib/kubelet/pods/3f440566-7401-430c-bfa0-e8de0007431b/volumes"
Sep 17 23:36:56.133461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.133437    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc45a81-340a-480c-9257-68ff117fea64-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:36:56.133461 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.133461    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8vzq\" (UniqueName: \"kubernetes.io/projected/4dc45a81-340a-480c-9257-68ff117fea64-kube-api-access-s8vzq\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:36:56.308455 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.308372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:56.424316 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.424277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:56.976109 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.976077    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"
Sep 17 23:36:56.993279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.993246    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"]
Sep 17 23:36:56.996284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:56.996261    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-145e15-7c64b75c46-hc4gm"]
Sep 17 23:36:57.125751 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:57.125718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:57.812970 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:57.812936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:58.109118 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:58.109038    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc45a81-340a-480c-9257-68ff117fea64" path="/var/lib/kubelet/pods/4dc45a81-340a-480c-9257-68ff117fea64/volumes"
Sep 17 23:36:58.577658 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:58.577629    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:59.229370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:59.229328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:59.433087 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:59.433052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:59.532856 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:36:59.532764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:02.027301 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:02.027263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:02.092611 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:02.092581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:02.263828 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:02.263780    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"]
Sep 17 23:37:02.264076 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:02.264051    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy" containerID="cri-o://dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397" gracePeriod=30
Sep 17 23:37:02.284556 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:02.284458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:04.730169 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:04.730130    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 17 23:37:04.730571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:04.730190    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:37:06.679321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:06.679288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:07.449822 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.449767    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:07.524518 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.524494    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:37:07.660420 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlpsb\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-kube-api-access-rlpsb\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-token\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-credential-socket\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-podinfo\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660547    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-data\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660619 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-socket\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-certs\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660668    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istiod-ca-cert\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-envoy\") pod \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\" (UID: \"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93\") "
Sep 17 23:37:07.660942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:07.660942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.660914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:07.661206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.661067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:07.661206 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.661107    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-credential-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.661483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.661204    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-data\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.661483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.661137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:07.661483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.661407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:07.662917 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.662894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-kube-api-access-rlpsb" (OuterVolumeSpecName: "kube-api-access-rlpsb") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "kube-api-access-rlpsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:07.663041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.662913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:07.663041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.662988    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 23:37:07.663041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.663002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" (UID: "ba32f8a2-fbc5-4914-a60c-f3ce755f1e93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:07.761963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.761914    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.761963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.761958    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-workload-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.761963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.761971    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istiod-ca-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.762411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.761984    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-envoy\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.762411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.761996    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlpsb\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-kube-api-access-rlpsb\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.762411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.762007    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-token\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:07.762411 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:07.762017    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93-istio-podinfo\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:08.038395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.038304    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerID="dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397" exitCode=0
Sep 17 23:37:08.038395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.038358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"
Sep 17 23:37:08.038698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.038392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" event={"ID":"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93","Type":"ContainerDied","Data":"dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397"}
Sep 17 23:37:08.038698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.038463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k" event={"ID":"ba32f8a2-fbc5-4914-a60c-f3ce755f1e93","Type":"ContainerDied","Data":"5cd314d03cb89af757415e034e4c70d0b90e84ddabfed1053560a1f705eb905d"}
Sep 17 23:37:08.038698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.038495    2580 scope.go:117] "RemoveContainer" containerID="dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397"
Sep 17 23:37:08.048836 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.048819    2580 scope.go:117] "RemoveContainer" containerID="64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7"
Sep 17 23:37:08.057523 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.057506    2580 scope.go:117] "RemoveContainer" containerID="dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397"
Sep 17 23:37:08.057876 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:37:08.057850    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397\": container with ID starting with dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397 not found: ID does not exist" containerID="dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397"
Sep 17 23:37:08.057959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.057888    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397"} err="failed to get container status \"dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397\": rpc error: code = NotFound desc = could not find container \"dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397\": container with ID starting with dbbfbb2c3df7c78ca54ec75dcfe0c391f0ace401eaa24e79bbc131c18e805397 not found: ID does not exist"
Sep 17 23:37:08.057959 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.057925    2580 scope.go:117] "RemoveContainer" containerID="64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7"
Sep 17 23:37:08.058645 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:37:08.058617    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7\": container with ID starting with 64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7 not found: ID does not exist" containerID="64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7"
Sep 17 23:37:08.058746 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.058653    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7"} err="failed to get container status \"64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7\": rpc error: code = NotFound desc = could not find container \"64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7\": container with ID starting with 64e700b2b39c66e47ce21c58105015553c8dbe226f1914839ae84ea3b55556d7 not found: ID does not exist"
Sep 17 23:37:08.060694 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.060672    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"]
Sep 17 23:37:08.064424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.064393    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-145e15-openshift-default-6c664977bb-hsg9k"]
Sep 17 23:37:08.108428 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.108398    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" path="/var/lib/kubelet/pods/ba32f8a2-fbc5-4914-a60c-f3ce755f1e93/volumes"
Sep 17 23:37:08.168166 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.168132    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:08.937296 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:08.937267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:09.657142 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:09.657114    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:21.372661 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.372628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:21.413765 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.413733    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"]
Sep 17 23:37:21.414069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.414037    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q" podUID="9bc0f952-4e2b-43df-95af-8eaab94aeec7" containerName="payload-pre-processing" containerID="cri-o://c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a" gracePeriod=30
Sep 17 23:37:21.671041 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.671011    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:37:21.697125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.697098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc0f952-4e2b-43df-95af-8eaab94aeec7-plugins-config-volume\") pod \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") "
Sep 17 23:37:21.697297 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.697231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb2px\" (UniqueName: \"kubernetes.io/projected/9bc0f952-4e2b-43df-95af-8eaab94aeec7-kube-api-access-mb2px\") pod \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\" (UID: \"9bc0f952-4e2b-43df-95af-8eaab94aeec7\") "
Sep 17 23:37:21.697492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.697469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc0f952-4e2b-43df-95af-8eaab94aeec7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bc0f952-4e2b-43df-95af-8eaab94aeec7" (UID: "9bc0f952-4e2b-43df-95af-8eaab94aeec7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:21.699549 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.699524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc0f952-4e2b-43df-95af-8eaab94aeec7-kube-api-access-mb2px" (OuterVolumeSpecName: "kube-api-access-mb2px") pod "9bc0f952-4e2b-43df-95af-8eaab94aeec7" (UID: "9bc0f952-4e2b-43df-95af-8eaab94aeec7"). InnerVolumeSpecName "kube-api-access-mb2px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:21.798995 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.798931    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2px\" (UniqueName: \"kubernetes.io/projected/9bc0f952-4e2b-43df-95af-8eaab94aeec7-kube-api-access-mb2px\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:21.798995 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:21.798986    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc0f952-4e2b-43df-95af-8eaab94aeec7-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.119479 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.119451    2580 generic.go:356] "Generic (PLEG): container finished" podID="9bc0f952-4e2b-43df-95af-8eaab94aeec7" containerID="c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a" exitCode=0
Sep 17 23:37:22.119605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.119498    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"
Sep 17 23:37:22.119605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.119526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q" event={"ID":"9bc0f952-4e2b-43df-95af-8eaab94aeec7","Type":"ContainerDied","Data":"c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a"}
Sep 17 23:37:22.119605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.119577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q" event={"ID":"9bc0f952-4e2b-43df-95af-8eaab94aeec7","Type":"ContainerDied","Data":"8455f326b1255c854a614545c79a8c99f13996813e1b675d7b3b9dc1367aff31"}
Sep 17 23:37:22.119605 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.119600    2580 scope.go:117] "RemoveContainer" containerID="c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a"
Sep 17 23:37:22.129240 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.129222    2580 scope.go:117] "RemoveContainer" containerID="c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a"
Sep 17 23:37:22.129529 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:37:22.129510    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a\": container with ID starting with c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a not found: ID does not exist" containerID="c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a"
Sep 17 23:37:22.129598 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.129542    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a"} err="failed to get container status \"c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a\": rpc error: code = NotFound desc = could not find container \"c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a\": container with ID starting with c023545f4aceb8128b00c161406076750aa2d1f71a66daa9f17da180681c9b7a not found: ID does not exist"
Sep 17 23:37:22.136371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.136348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"]
Sep 17 23:37:22.143757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.143734    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93a496-66987d7998-6tx7q"]
Sep 17 23:37:22.883314 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:22.883278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:23.434702 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:23.434674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:24.108253 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:24.108222    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc0f952-4e2b-43df-95af-8eaab94aeec7" path="/var/lib/kubelet/pods/9bc0f952-4e2b-43df-95af-8eaab94aeec7/volumes"
Sep 17 23:37:24.125579 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:24.125543    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:26.929451 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:26.929417    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"]
Sep 17 23:37:26.983131 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:26.983093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:30.430174 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:30.430130    2580 scope.go:117] "RemoveContainer" containerID="7925be1152308af565631b880e641f8f220d5915e413fa1bfb3a74470bf692c0"
Sep 17 23:37:30.438637 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:30.438617    2580 scope.go:117] "RemoveContainer" containerID="1bc697abed33d85f793e74ed292a3ae43a0e5e311e2da4a4b89461391234dd4b"
Sep 17 23:37:41.953229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:41.953157    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4" podUID="697cfe44-a8d6-46ce-80f3-cfbf0fa04086" containerName="main" containerID="cri-o://78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9" gracePeriod=45
Sep 17 23:37:42.207005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.206932    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:37:42.230754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.230625    2580 generic.go:356] "Generic (PLEG): container finished" podID="697cfe44-a8d6-46ce-80f3-cfbf0fa04086" containerID="78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9" exitCode=0
Sep 17 23:37:42.230754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.230666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4" event={"ID":"697cfe44-a8d6-46ce-80f3-cfbf0fa04086","Type":"ContainerDied","Data":"78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9"}
Sep 17 23:37:42.230754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.230685    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"
Sep 17 23:37:42.230754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.230706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4" event={"ID":"697cfe44-a8d6-46ce-80f3-cfbf0fa04086","Type":"ContainerDied","Data":"1b242e8266784544d25f85e23dc4393bcb3f489ef9d940890599c9cd0f99a05e"}
Sep 17 23:37:42.230754 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.230727    2580 scope.go:117] "RemoveContainer" containerID="78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9"
Sep 17 23:37:42.242213 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.242189    2580 scope.go:117] "RemoveContainer" containerID="78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9"
Sep 17 23:37:42.242568 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:37:42.242540    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9\": container with ID starting with 78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9 not found: ID does not exist" containerID="78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9"
Sep 17 23:37:42.242662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.242583    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9"} err="failed to get container status \"78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9\": rpc error: code = NotFound desc = could not find container \"78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9\": container with ID starting with 78cfa01bcb30dd00bed94036981b3f1153e6e1b2675b345c887ec16b9bd5b0b9 not found: ID does not exist"
Sep 17 23:37:42.301847 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.301786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tmp-dir\") pod \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") "
Sep 17 23:37:42.302058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.301859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-dshm\") pod \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") "
Sep 17 23:37:42.302058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.301906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tls-certs\") pod \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") "
Sep 17 23:37:42.302058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.301957    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-model-cache\") pod \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") "
Sep 17 23:37:42.302058 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6whjl\" (UniqueName: \"kubernetes.io/projected/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-kube-api-access-6whjl\") pod \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") "
Sep 17 23:37:42.302272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-home\") pod \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\" (UID: \"697cfe44-a8d6-46ce-80f3-cfbf0fa04086\") "
Sep 17 23:37:42.302272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302090    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "697cfe44-a8d6-46ce-80f3-cfbf0fa04086" (UID: "697cfe44-a8d6-46ce-80f3-cfbf0fa04086"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:42.302272 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-model-cache" (OuterVolumeSpecName: "model-cache") pod "697cfe44-a8d6-46ce-80f3-cfbf0fa04086" (UID: "697cfe44-a8d6-46ce-80f3-cfbf0fa04086"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:42.302404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-home" (OuterVolumeSpecName: "home") pod "697cfe44-a8d6-46ce-80f3-cfbf0fa04086" (UID: "697cfe44-a8d6-46ce-80f3-cfbf0fa04086"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:42.302467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302425    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:42.302467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302444    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:42.302467 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.302457    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:42.304338 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.304309    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-dshm" (OuterVolumeSpecName: "dshm") pod "697cfe44-a8d6-46ce-80f3-cfbf0fa04086" (UID: "697cfe44-a8d6-46ce-80f3-cfbf0fa04086"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:42.304452 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.304354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "697cfe44-a8d6-46ce-80f3-cfbf0fa04086" (UID: "697cfe44-a8d6-46ce-80f3-cfbf0fa04086"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:42.304452 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.304371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-kube-api-access-6whjl" (OuterVolumeSpecName: "kube-api-access-6whjl") pod "697cfe44-a8d6-46ce-80f3-cfbf0fa04086" (UID: "697cfe44-a8d6-46ce-80f3-cfbf0fa04086"). InnerVolumeSpecName "kube-api-access-6whjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:42.403394 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.403354    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:42.403394 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.403387    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:42.403394 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.403401    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whjl\" (UniqueName: \"kubernetes.io/projected/697cfe44-a8d6-46ce-80f3-cfbf0fa04086-kube-api-access-6whjl\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:37:42.551974 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.551944    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"]
Sep 17 23:37:42.556137 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:42.556113    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-93a496/ipp-route-93a496-kserve-f8cf79c9-tzqh4"]
Sep 17 23:37:44.107977 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:37:44.107935    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697cfe44-a8d6-46ce-80f3-cfbf0fa04086" path="/var/lib/kubelet/pods/697cfe44-a8d6-46ce-80f3-cfbf0fa04086/volumes"
Sep 17 23:38:03.975878 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:03.975729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:04.391422 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:04.391368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:10.160384 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.160345    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:10.222857 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.222748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:10.439374 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.439284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:10.811280 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811196    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"]
Sep 17 23:38:10.811602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811589    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy"
Sep 17 23:38:10.811652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811603    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy"
Sep 17 23:38:10.811652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811615    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy"
Sep 17 23:38:10.811652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811620    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy"
Sep 17 23:38:10.811652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811635    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc45a81-340a-480c-9257-68ff117fea64" containerName="payload-pre-processing"
Sep 17 23:38:10.811652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811640    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc45a81-340a-480c-9257-68ff117fea64" containerName="payload-pre-processing"
Sep 17 23:38:10.811652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811650    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697cfe44-a8d6-46ce-80f3-cfbf0fa04086" containerName="main"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="697cfe44-a8d6-46ce-80f3-cfbf0fa04086" containerName="main"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f440566-7401-430c-bfa0-e8de0007431b" containerName="revoke-keys"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811672    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f440566-7401-430c-bfa0-e8de0007431b" containerName="revoke-keys"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811681    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc0f952-4e2b-43df-95af-8eaab94aeec7" containerName="payload-pre-processing"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811687    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc0f952-4e2b-43df-95af-8eaab94aeec7" containerName="payload-pre-processing"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811742    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811750    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba32f8a2-fbc5-4914-a60c-f3ce755f1e93" containerName="istio-proxy"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811758    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="697cfe44-a8d6-46ce-80f3-cfbf0fa04086" containerName="main"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811765    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc45a81-340a-480c-9257-68ff117fea64" containerName="payload-pre-processing"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811771    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f440566-7401-430c-bfa0-e8de0007431b" containerName="revoke-keys"
Sep 17 23:38:10.811882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.811776    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc0f952-4e2b-43df-95af-8eaab94aeec7" containerName="payload-pre-processing"
Sep 17 23:38:10.814289 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.814273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"
Sep 17 23:38:10.816503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.816480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:38:10.816616 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.816509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:38:10.821005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.820978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"]
Sep 17 23:38:10.878069 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.878027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ebe8b-1445-4b2c-84bc-c54a3f38c193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj\" (UID: \"203ebe8b-1445-4b2c-84bc-c54a3f38c193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"
Sep 17 23:38:10.979036 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.978999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ebe8b-1445-4b2c-84bc-c54a3f38c193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj\" (UID: \"203ebe8b-1445-4b2c-84bc-c54a3f38c193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"
Sep 17 23:38:10.979740 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:10.979683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ebe8b-1445-4b2c-84bc-c54a3f38c193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj\" (UID: \"203ebe8b-1445-4b2c-84bc-c54a3f38c193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"
Sep 17 23:38:11.069326 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:11.069232    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:11.125464 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:11.125427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"
Sep 17 23:38:11.398715 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:11.398691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"]
Sep 17 23:38:11.400558 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:38:11.400536    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203ebe8b_1445_4b2c_84bc_c54a3f38c193.slice/crio-78affe2631e31c283f923461c21a00782a3a72a059c72f88a6ae8819142c76d4 WatchSource:0}: Error finding container 78affe2631e31c283f923461c21a00782a3a72a059c72f88a6ae8819142c76d4: Status 404 returned error can't find the container with id 78affe2631e31c283f923461c21a00782a3a72a059c72f88a6ae8819142c76d4
Sep 17 23:38:12.147854 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:12.147792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:12.399680 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:12.399591    2580 generic.go:356] "Generic (PLEG): container finished" podID="203ebe8b-1445-4b2c-84bc-c54a3f38c193" containerID="2322f05203954c670f8da1d1fa585c336e6343fc97afb1d2d0042c82f40968df" exitCode=0
Sep 17 23:38:12.400054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:12.399685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj" event={"ID":"203ebe8b-1445-4b2c-84bc-c54a3f38c193","Type":"ContainerDied","Data":"2322f05203954c670f8da1d1fa585c336e6343fc97afb1d2d0042c82f40968df"}
Sep 17 23:38:12.400054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:12.399745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj" event={"ID":"203ebe8b-1445-4b2c-84bc-c54a3f38c193","Type":"ContainerStarted","Data":"78affe2631e31c283f923461c21a00782a3a72a059c72f88a6ae8819142c76d4"}
Sep 17 23:38:13.309196 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:13.309163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:13.533553 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:13.533530    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"
Sep 17 23:38:13.604102 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:13.604012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ebe8b-1445-4b2c-84bc-c54a3f38c193-maas-api-service-ca\") pod \"203ebe8b-1445-4b2c-84bc-c54a3f38c193\" (UID: \"203ebe8b-1445-4b2c-84bc-c54a3f38c193\") "
Sep 17 23:38:13.604438 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:13.604405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203ebe8b-1445-4b2c-84bc-c54a3f38c193-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "203ebe8b-1445-4b2c-84bc-c54a3f38c193" (UID: "203ebe8b-1445-4b2c-84bc-c54a3f38c193"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:13.705381 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:13.705347    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ebe8b-1445-4b2c-84bc-c54a3f38c193-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:14.345026 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:14.344987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:14.412588 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:14.412561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"
Sep 17 23:38:14.412757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:14.412558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj" event={"ID":"203ebe8b-1445-4b2c-84bc-c54a3f38c193","Type":"ContainerDied","Data":"78affe2631e31c283f923461c21a00782a3a72a059c72f88a6ae8819142c76d4"}
Sep 17 23:38:14.412757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:14.412684    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78affe2631e31c283f923461c21a00782a3a72a059c72f88a6ae8819142c76d4"
Sep 17 23:38:15.148433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:15.148396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:21.252669 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.252635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:21.283338 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.283302    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"]
Sep 17 23:38:21.283618 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.283585    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6" podUID="a6e4706c-7b4e-4b6c-8478-daa1618b144d" containerName="payload-pre-processing" containerID="cri-o://0be8e21bfec540b5c2d8dd722332ea38687aa0b27753dee48921147971b8830b" gracePeriod=30
Sep 17 23:38:21.352561 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.352528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:21.431904 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.431861    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"]
Sep 17 23:38:21.434177 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.434143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:21.434511 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.434492    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86dda1-2t2nj"]
Sep 17 23:38:21.453935 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.453909    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6e4706c-7b4e-4b6c-8478-daa1618b144d" containerID="0be8e21bfec540b5c2d8dd722332ea38687aa0b27753dee48921147971b8830b" exitCode=0
Sep 17 23:38:21.454050 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.453950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6" event={"ID":"a6e4706c-7b4e-4b6c-8478-daa1618b144d","Type":"ContainerDied","Data":"0be8e21bfec540b5c2d8dd722332ea38687aa0b27753dee48921147971b8830b"}
Sep 17 23:38:21.499860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.499821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:21.550523 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.550499    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:38:21.682679 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.682636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e4706c-7b4e-4b6c-8478-daa1618b144d-plugins-config-volume\") pod \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") "
Sep 17 23:38:21.682892 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.682713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4kkx\" (UniqueName: \"kubernetes.io/projected/a6e4706c-7b4e-4b6c-8478-daa1618b144d-kube-api-access-n4kkx\") pod \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\" (UID: \"a6e4706c-7b4e-4b6c-8478-daa1618b144d\") "
Sep 17 23:38:21.683090 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.683069    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e4706c-7b4e-4b6c-8478-daa1618b144d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6e4706c-7b4e-4b6c-8478-daa1618b144d" (UID: "a6e4706c-7b4e-4b6c-8478-daa1618b144d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:21.683193 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.683180    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e4706c-7b4e-4b6c-8478-daa1618b144d-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:21.685008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.684986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e4706c-7b4e-4b6c-8478-daa1618b144d-kube-api-access-n4kkx" (OuterVolumeSpecName: "kube-api-access-n4kkx") pod "a6e4706c-7b4e-4b6c-8478-daa1618b144d" (UID: "a6e4706c-7b4e-4b6c-8478-daa1618b144d"). InnerVolumeSpecName "kube-api-access-n4kkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:21.784409 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.784317    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4kkx\" (UniqueName: \"kubernetes.io/projected/a6e4706c-7b4e-4b6c-8478-daa1618b144d-kube-api-access-n4kkx\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:21.831089 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.831054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:21.931545 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:21.931512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:22.115305 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:22.109408    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203ebe8b-1445-4b2c-84bc-c54a3f38c193" path="/var/lib/kubelet/pods/203ebe8b-1445-4b2c-84bc-c54a3f38c193/volumes"
Sep 17 23:38:22.459971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:22.459933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6" event={"ID":"a6e4706c-7b4e-4b6c-8478-daa1618b144d","Type":"ContainerDied","Data":"27b83271960e53c7b54a912d050f9876e64616870a074785022ba60d5c2dc833"}
Sep 17 23:38:22.459971 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:22.459971    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"
Sep 17 23:38:22.460583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:22.459996    2580 scope.go:117] "RemoveContainer" containerID="0be8e21bfec540b5c2d8dd722332ea38687aa0b27753dee48921147971b8830b"
Sep 17 23:38:22.474739 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:22.474710    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"]
Sep 17 23:38:22.483171 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:22.483140    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86dda1-69c86d65bb-24ww6"]
Sep 17 23:38:22.572538 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:22.572502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:23.331965 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:23.331926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:24.108662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:24.108630    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e4706c-7b4e-4b6c-8478-daa1618b144d" path="/var/lib/kubelet/pods/a6e4706c-7b4e-4b6c-8478-daa1618b144d/volumes"
Sep 17 23:38:27.897739 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:27.897711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:27.965254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:27.965215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:28.133185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.133151    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"]
Sep 17 23:38:28.133391 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.133370    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy" containerID="cri-o://03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02" gracePeriod=30
Sep 17 23:38:28.159849 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.159777    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:28.505512 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.505422    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"]
Sep 17 23:38:28.505825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.505788    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e4706c-7b4e-4b6c-8478-daa1618b144d" containerName="payload-pre-processing"
Sep 17 23:38:28.505825 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.505816    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e4706c-7b4e-4b6c-8478-daa1618b144d" containerName="payload-pre-processing"
Sep 17 23:38:28.505912 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.505837    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203ebe8b-1445-4b2c-84bc-c54a3f38c193" containerName="revoke-keys"
Sep 17 23:38:28.505912 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.505843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="203ebe8b-1445-4b2c-84bc-c54a3f38c193" containerName="revoke-keys"
Sep 17 23:38:28.505984 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.505912    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="203ebe8b-1445-4b2c-84bc-c54a3f38c193" containerName="revoke-keys"
Sep 17 23:38:28.505984 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.505922    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e4706c-7b4e-4b6c-8478-daa1618b144d" containerName="payload-pre-processing"
Sep 17 23:38:28.510215 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.510196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"
Sep 17 23:38:28.512767 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.512745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:38:28.512896 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.512826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:38:28.515786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.515761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"]
Sep 17 23:38:28.652244 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.652210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5134b78-a900-4bcf-b7ab-7222692756aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2\" (UID: \"f5134b78-a900-4bcf-b7ab-7222692756aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"
Sep 17 23:38:28.753431 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.753388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5134b78-a900-4bcf-b7ab-7222692756aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2\" (UID: \"f5134b78-a900-4bcf-b7ab-7222692756aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"
Sep 17 23:38:28.754055 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.754035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5134b78-a900-4bcf-b7ab-7222692756aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2\" (UID: \"f5134b78-a900-4bcf-b7ab-7222692756aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"
Sep 17 23:38:28.823549 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:28.823455    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"
Sep 17 23:38:29.095492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:29.095326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"]
Sep 17 23:38:29.097967 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:38:29.097941    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5134b78_a900_4bcf_b7ab_7222692756aa.slice/crio-137ab55c9c9e1b4b2df17227af39fd1f68f748e5a39680b79086c6d409788572 WatchSource:0}: Error finding container 137ab55c9c9e1b4b2df17227af39fd1f68f748e5a39680b79086c6d409788572: Status 404 returned error can't find the container with id 137ab55c9c9e1b4b2df17227af39fd1f68f748e5a39680b79086c6d409788572
Sep 17 23:38:29.248531 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:29.248493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:29.500775 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:29.500738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2" event={"ID":"f5134b78-a900-4bcf-b7ab-7222692756aa","Type":"ContainerStarted","Data":"d40c01cc5fca49b4764ee66269af4422c1ffb09cda3c3408bf1632e564baf6a2"}
Sep 17 23:38:29.500969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:29.500816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2" event={"ID":"f5134b78-a900-4bcf-b7ab-7222692756aa","Type":"ContainerStarted","Data":"137ab55c9c9e1b4b2df17227af39fd1f68f748e5a39680b79086c6d409788572"}
Sep 17 23:38:29.547514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:29.547455    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2" podStartSLOduration=1.5474374480000002 podStartE2EDuration="1.547437448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:29.546122306 +0000 UTC m=+1919.975982124" watchObservedRunningTime="2026-09-17 23:38:29.547437448 +0000 UTC m=+1919.977297265"
Sep 17 23:38:30.150129 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:30.150093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:30.507722 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:30.507692    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5134b78-a900-4bcf-b7ab-7222692756aa" containerID="d40c01cc5fca49b4764ee66269af4422c1ffb09cda3c3408bf1632e564baf6a2" exitCode=0
Sep 17 23:38:30.507850 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:30.507772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2" event={"ID":"f5134b78-a900-4bcf-b7ab-7222692756aa","Type":"ContainerDied","Data":"d40c01cc5fca49b4764ee66269af4422c1ffb09cda3c3408bf1632e564baf6a2"}
Sep 17 23:38:30.807426 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:30.807344    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:31.643839 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:31.643816    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"
Sep 17 23:38:31.679977 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:31.679953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5134b78-a900-4bcf-b7ab-7222692756aa-maas-api-service-ca\") pod \"f5134b78-a900-4bcf-b7ab-7222692756aa\" (UID: \"f5134b78-a900-4bcf-b7ab-7222692756aa\") "
Sep 17 23:38:31.680493 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:31.680464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5134b78-a900-4bcf-b7ab-7222692756aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5134b78-a900-4bcf-b7ab-7222692756aa" (UID: "f5134b78-a900-4bcf-b7ab-7222692756aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:31.781142 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:31.781054    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5134b78-a900-4bcf-b7ab-7222692756aa-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:32.519672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:32.519641    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"
Sep 17 23:38:32.519672 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:32.519656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2" event={"ID":"f5134b78-a900-4bcf-b7ab-7222692756aa","Type":"ContainerDied","Data":"137ab55c9c9e1b4b2df17227af39fd1f68f748e5a39680b79086c6d409788572"}
Sep 17 23:38:32.519906 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:32.519696    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="137ab55c9c9e1b4b2df17227af39fd1f68f748e5a39680b79086c6d409788572"
Sep 17 23:38:33.387302 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.387278    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:38:33.396062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-token\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-podinfo\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-socket\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-credential-socket\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396152    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756fd94-7c01-40b2-96f3-098a0cce75dd-istiod-ca-cert\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-certs\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-data\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6frg\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-kube-api-access-q6frg\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396268 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-envoy\") pod \"4756fd94-7c01-40b2-96f3-098a0cce75dd\" (UID: \"4756fd94-7c01-40b2-96f3-098a0cce75dd\") "
Sep 17 23:38:33.396643 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:33.396643 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396623    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:33.396934 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:33.397003 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.396946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4756fd94-7c01-40b2-96f3-098a0cce75dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:33.397246 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.397220    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:33.398453 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.398425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:33.398565 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.398514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 23:38:33.398718 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.398697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:33.398956 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.398926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-kube-api-access-q6frg" (OuterVolumeSpecName: "kube-api-access-q6frg") pod "4756fd94-7c01-40b2-96f3-098a0cce75dd" (UID: "4756fd94-7c01-40b2-96f3-098a0cce75dd"). InnerVolumeSpecName "kube-api-access-q6frg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:33.497818 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497774    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497837    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-credential-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497849    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4756fd94-7c01-40b2-96f3-098a0cce75dd-istiod-ca-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497858    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-workload-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497865    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-data\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497873    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6frg\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-kube-api-access-q6frg\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497881    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-envoy\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497892    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-token\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.497992 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.497904    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4756fd94-7c01-40b2-96f3-098a0cce75dd-istio-podinfo\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:33.527404 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.527373    2580 generic.go:356] "Generic (PLEG): container finished" podID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerID="03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02" exitCode=0
Sep 17 23:38:33.527575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.527419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"
Sep 17 23:38:33.527575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.527447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" event={"ID":"4756fd94-7c01-40b2-96f3-098a0cce75dd","Type":"ContainerDied","Data":"03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02"}
Sep 17 23:38:33.527575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.527510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2" event={"ID":"4756fd94-7c01-40b2-96f3-098a0cce75dd","Type":"ContainerDied","Data":"24e869de348586b2b4d6871de51e4e540011ec4e9d8e1ad13c8ef3dc7aeb8e40"}
Sep 17 23:38:33.527575 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.527539    2580 scope.go:117] "RemoveContainer" containerID="03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02"
Sep 17 23:38:33.537707 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.537690    2580 scope.go:117] "RemoveContainer" containerID="e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73"
Sep 17 23:38:33.545932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.545917    2580 scope.go:117] "RemoveContainer" containerID="03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02"
Sep 17 23:38:33.546201 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:38:33.546182    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02\": container with ID starting with 03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02 not found: ID does not exist" containerID="03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02"
Sep 17 23:38:33.546254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.546210    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02"} err="failed to get container status \"03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02\": rpc error: code = NotFound desc = could not find container \"03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02\": container with ID starting with 03a24c680fae6f38592048f896eff6e585ca77575d36e239681b10aa48e37e02 not found: ID does not exist"
Sep 17 23:38:33.546254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.546227    2580 scope.go:117] "RemoveContainer" containerID="e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73"
Sep 17 23:38:33.546483 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:38:33.546462    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73\": container with ID starting with e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73 not found: ID does not exist" containerID="e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73"
Sep 17 23:38:33.546585 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.546485    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73"} err="failed to get container status \"e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73\": rpc error: code = NotFound desc = could not find container \"e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73\": container with ID starting with e13d70130f3ed68d5fcf29a1e62ecfa672acbfc43acb9e427ead4cf55cbb9d73 not found: ID does not exist"
Sep 17 23:38:33.549224 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.549202    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"]
Sep 17 23:38:33.553510 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:33.553488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dda1-openshift-default-67b54b6584-kcvr2"]
Sep 17 23:38:34.108448 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:34.108415    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" path="/var/lib/kubelet/pods/4756fd94-7c01-40b2-96f3-098a0cce75dd/volumes"
Sep 17 23:38:36.572988 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:36.572958    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:36.640871 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:36.640835    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:39.054571 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.054528    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"]
Sep 17 23:38:39.057072 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.057048    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e5aae7-29828130-gsg6x"]
Sep 17 23:38:39.060940 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.060916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:39.084481 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.084449    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"]
Sep 17 23:38:39.084795 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.084762    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs" podUID="323da953-19d0-4aa5-8d2a-d25f1ccc71e9" containerName="payload-processing" containerID="cri-o://c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d" gracePeriod=30
Sep 17 23:38:39.127929 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.127896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:39.227484 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.227446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:39.235905 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.235578    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"]
Sep 17 23:38:39.237449 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.237428    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-e5aae7-shqz2"]
Sep 17 23:38:39.293216 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.293186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:39.336536 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.336512    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:38:39.452978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.452947    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-plugins-config-volume\") pod \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") "
Sep 17 23:38:39.452978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.452981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvs2m\" (UniqueName: \"kubernetes.io/projected/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-kube-api-access-bvs2m\") pod \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\" (UID: \"323da953-19d0-4aa5-8d2a-d25f1ccc71e9\") "
Sep 17 23:38:39.453369 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.453349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "323da953-19d0-4aa5-8d2a-d25f1ccc71e9" (UID: "323da953-19d0-4aa5-8d2a-d25f1ccc71e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:39.455254 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.455231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-kube-api-access-bvs2m" (OuterVolumeSpecName: "kube-api-access-bvs2m") pod "323da953-19d0-4aa5-8d2a-d25f1ccc71e9" (UID: "323da953-19d0-4aa5-8d2a-d25f1ccc71e9"). InnerVolumeSpecName "kube-api-access-bvs2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:39.554115 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.554080    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-plugins-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:39.554115 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.554113    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvs2m\" (UniqueName: \"kubernetes.io/projected/323da953-19d0-4aa5-8d2a-d25f1ccc71e9-kube-api-access-bvs2m\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:39.559496 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.559465    2580 generic.go:356] "Generic (PLEG): container finished" podID="323da953-19d0-4aa5-8d2a-d25f1ccc71e9" containerID="c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d" exitCode=0
Sep 17 23:38:39.559668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.559513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs" event={"ID":"323da953-19d0-4aa5-8d2a-d25f1ccc71e9","Type":"ContainerDied","Data":"c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d"}
Sep 17 23:38:39.559668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.559520    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"
Sep 17 23:38:39.559668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.559567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs" event={"ID":"323da953-19d0-4aa5-8d2a-d25f1ccc71e9","Type":"ContainerDied","Data":"cc42f48d48c3c14262fc3204b8c98022bf98c5142497a672f10eb96e68780567"}
Sep 17 23:38:39.559668 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.559599    2580 scope.go:117] "RemoveContainer" containerID="c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d"
Sep 17 23:38:39.570665 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.570428    2580 scope.go:117] "RemoveContainer" containerID="c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d"
Sep 17 23:38:39.570997 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:38:39.570915    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d\": container with ID starting with c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d not found: ID does not exist" containerID="c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d"
Sep 17 23:38:39.570997 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.570968    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d"} err="failed to get container status \"c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d\": rpc error: code = NotFound desc = could not find container \"c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d\": container with ID starting with c5a3cc6d9d9fd4bbde1b344cbd881b14c00bd930d406368f6ff5bf52a820d41d not found: ID does not exist"
Sep 17 23:38:39.582662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.582636    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"]
Sep 17 23:38:39.589604 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.589580    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e5aae7-574b759d99-pr2qs"]
Sep 17 23:38:39.607791 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.607764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:39.710405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:39.710365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:40.108071 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:40.108033    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274" path="/var/lib/kubelet/pods/02a4f5ee-f1c7-4e2d-8343-8a8bd2dc5274/volumes"
Sep 17 23:38:40.108448 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:40.108362    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="323da953-19d0-4aa5-8d2a-d25f1ccc71e9" path="/var/lib/kubelet/pods/323da953-19d0-4aa5-8d2a-d25f1ccc71e9/volumes"
Sep 17 23:38:40.108647 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:40.108635    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5134b78-a900-4bcf-b7ab-7222692756aa" path="/var/lib/kubelet/pods/f5134b78-a900-4bcf-b7ab-7222692756aa/volumes"
Sep 17 23:38:40.438123 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:40.438085    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:45.736492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:45.736448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:45.838298 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:45.838256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:45.987643 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:45.987564    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:46.377116 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377028    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"]
Sep 17 23:38:46.377395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377383    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5134b78-a900-4bcf-b7ab-7222692756aa" containerName="revoke-keys"
Sep 17 23:38:46.377440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377397    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5134b78-a900-4bcf-b7ab-7222692756aa" containerName="revoke-keys"
Sep 17 23:38:46.377440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377410    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy"
Sep 17 23:38:46.377440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377416    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy"
Sep 17 23:38:46.377440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377425    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323da953-19d0-4aa5-8d2a-d25f1ccc71e9" containerName="payload-processing"
Sep 17 23:38:46.377440 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377431    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="323da953-19d0-4aa5-8d2a-d25f1ccc71e9" containerName="payload-processing"
Sep 17 23:38:46.377592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377443    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy"
Sep 17 23:38:46.377592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377449    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy"
Sep 17 23:38:46.377592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377512    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="323da953-19d0-4aa5-8d2a-d25f1ccc71e9" containerName="payload-processing"
Sep 17 23:38:46.377592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377521    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy"
Sep 17 23:38:46.377592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.377529    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5134b78-a900-4bcf-b7ab-7222692756aa" containerName="revoke-keys"
Sep 17 23:38:46.381670 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.381652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"
Sep 17 23:38:46.384287 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.384263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5ldj\""
Sep 17 23:38:46.384742 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.384724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:38:46.390062 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.390004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"]
Sep 17 23:38:46.521488 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.521455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a60ef7-d072-452a-b6c4-97022e2e94e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84\" (UID: \"a4a60ef7-d072-452a-b6c4-97022e2e94e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"
Sep 17 23:38:46.622652 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.622618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a60ef7-d072-452a-b6c4-97022e2e94e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84\" (UID: \"a4a60ef7-d072-452a-b6c4-97022e2e94e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"
Sep 17 23:38:46.623279 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.623260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a60ef7-d072-452a-b6c4-97022e2e94e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84\" (UID: \"a4a60ef7-d072-452a-b6c4-97022e2e94e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"
Sep 17 23:38:46.649366 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.649304    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:46.696365 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.696332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"
Sep 17 23:38:46.970293 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:46.970266    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"]
Sep 17 23:38:46.973067 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:38:46.973043    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a60ef7_d072_452a_b6c4_97022e2e94e4.slice/crio-3c4a9e18821023ca292d48ab956d5e3f23d749634d423e6ee8c2d16e13e11abd WatchSource:0}: Error finding container 3c4a9e18821023ca292d48ab956d5e3f23d749634d423e6ee8c2d16e13e11abd: Status 404 returned error can't find the container with id 3c4a9e18821023ca292d48ab956d5e3f23d749634d423e6ee8c2d16e13e11abd
Sep 17 23:38:47.452982 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:47.452943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:47.605637 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:47.605545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84" event={"ID":"a4a60ef7-d072-452a-b6c4-97022e2e94e4","Type":"ContainerStarted","Data":"8347bcdc390851ae2cda6a56d2b64f052a619bcf4b57cfb48a4d13716dd3205d"}
Sep 17 23:38:47.605637 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:47.605601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84" event={"ID":"a4a60ef7-d072-452a-b6c4-97022e2e94e4","Type":"ContainerStarted","Data":"3c4a9e18821023ca292d48ab956d5e3f23d749634d423e6ee8c2d16e13e11abd"}
Sep 17 23:38:47.623196 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:47.623145    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84" podStartSLOduration=1.62312982 podStartE2EDuration="1.62312982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:47.620983194 +0000 UTC m=+1938.050843011" watchObservedRunningTime="2026-09-17 23:38:47.62312982 +0000 UTC m=+1938.052989638"
Sep 17 23:38:48.234253 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:48.234212    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:48.612554 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:48.612469    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4a60ef7-d072-452a-b6c4-97022e2e94e4" containerID="8347bcdc390851ae2cda6a56d2b64f052a619bcf4b57cfb48a4d13716dd3205d" exitCode=0
Sep 17 23:38:48.612554 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:48.612531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84" event={"ID":"a4a60ef7-d072-452a-b6c4-97022e2e94e4","Type":"ContainerDied","Data":"8347bcdc390851ae2cda6a56d2b64f052a619bcf4b57cfb48a4d13716dd3205d"}
Sep 17 23:38:49.754182 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:49.754156    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"
Sep 17 23:38:49.852880 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:49.852848    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a60ef7-d072-452a-b6c4-97022e2e94e4-maas-api-service-ca\") pod \"a4a60ef7-d072-452a-b6c4-97022e2e94e4\" (UID: \"a4a60ef7-d072-452a-b6c4-97022e2e94e4\") "
Sep 17 23:38:49.853235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:49.853209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a60ef7-d072-452a-b6c4-97022e2e94e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4a60ef7-d072-452a-b6c4-97022e2e94e4" (UID: "a4a60ef7-d072-452a-b6c4-97022e2e94e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:49.954256 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:49.954220    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4a60ef7-d072-452a-b6c4-97022e2e94e4-maas-api-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:38:50.622991 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:50.622958    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"
Sep 17 23:38:50.623186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:50.622957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84" event={"ID":"a4a60ef7-d072-452a-b6c4-97022e2e94e4","Type":"ContainerDied","Data":"3c4a9e18821023ca292d48ab956d5e3f23d749634d423e6ee8c2d16e13e11abd"}
Sep 17 23:38:50.623186 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:50.623077    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c4a9e18821023ca292d48ab956d5e3f23d749634d423e6ee8c2d16e13e11abd"
Sep 17 23:38:56.901323 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:56.901280    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"]
Sep 17 23:38:56.906275 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:56.906248    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6275e9-29828130-mpn67"]
Sep 17 23:38:56.909756 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:56.909712    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:56.979858 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:56.979779    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:57.079865 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:57.079826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:57.108990 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:57.108941    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"]
Sep 17 23:38:57.110932 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:57.110903    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6275e9-f7x84"]
Sep 17 23:38:57.147551 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:57.147521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:57.490622 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:57.490580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:57.558592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:57.558561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:58.108904 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:58.108871    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0b925b-23a3-45c7-a37f-ece41272acd7" path="/var/lib/kubelet/pods/7f0b925b-23a3-45c7-a37f-ece41272acd7/volumes"
Sep 17 23:38:58.109278 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:38:58.109201    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a60ef7-d072-452a-b6c4-97022e2e94e4" path="/var/lib/kubelet/pods/a4a60ef7-d072-452a-b6c4-97022e2e94e4/volumes"
Sep 17 23:39:03.444057 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:03.444019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:03.606052 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:03.606019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:03.689835 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:03.689787    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"]
Sep 17 23:39:03.690088 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:03.690063    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy" containerID="cri-o://150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b" gracePeriod=30
Sep 17 23:39:03.713979 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:03.713903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:08.938084 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:08.938055    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:39:09.033274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033177    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-socket\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-credential-socket\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-token\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033302    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-certs\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-data\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-podinfo\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glqtv\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-kube-api-access-glqtv\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-envoy\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b101bd-be25-4f15-b585-5f45a64b1dcd-istiod-ca-cert\") pod \"32b101bd-be25-4f15-b585-5f45a64b1dcd\" (UID: \"32b101bd-be25-4f15-b585-5f45a64b1dcd\") "
Sep 17 23:39:09.033911 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:09.033911 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:09.033911 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:09.033911 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.033737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-data" (OuterVolumeSpecName: "istio-data") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:09.034229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.034168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b101bd-be25-4f15-b585-5f45a64b1dcd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:09.035791 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.035761    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-token" (OuterVolumeSpecName: "istio-token") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:09.035925 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.035817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-kube-api-access-glqtv" (OuterVolumeSpecName: "kube-api-access-glqtv") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "kube-api-access-glqtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:09.036273 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.036250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:09.036273 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.036251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32b101bd-be25-4f15-b585-5f45a64b1dcd" (UID: "32b101bd-be25-4f15-b585-5f45a64b1dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 23:39:09.134814 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134760    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-data\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.134814 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134821    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-podinfo\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.135027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134838    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqtv\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-kube-api-access-glqtv\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.135027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134853    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-envoy\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.135027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134866    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b101bd-be25-4f15-b585-5f45a64b1dcd-istiod-ca-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.135027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134878    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.135027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134890    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-credential-socket\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.135027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134902    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b101bd-be25-4f15-b585-5f45a64b1dcd-istio-token\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.135027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.134915    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b101bd-be25-4f15-b585-5f45a64b1dcd-workload-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:39:09.732249 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.732218    2580 generic.go:356] "Generic (PLEG): container finished" podID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerID="150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b" exitCode=0
Sep 17 23:39:09.732514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.732263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" event={"ID":"32b101bd-be25-4f15-b585-5f45a64b1dcd","Type":"ContainerDied","Data":"150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b"}
Sep 17 23:39:09.732514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.732277    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"
Sep 17 23:39:09.732514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.732306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7" event={"ID":"32b101bd-be25-4f15-b585-5f45a64b1dcd","Type":"ContainerDied","Data":"96984f6467ff269f7fe91e5b742f1707a7dba398278c0005494a38a29b20442a"}
Sep 17 23:39:09.732514 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.732327    2580 scope.go:117] "RemoveContainer" containerID="150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b"
Sep 17 23:39:09.744054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.744027    2580 scope.go:117] "RemoveContainer" containerID="de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f"
Sep 17 23:39:09.752851 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.752789    2580 scope.go:117] "RemoveContainer" containerID="150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b"
Sep 17 23:39:09.753224 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:39:09.753192    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b\": container with ID starting with 150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b not found: ID does not exist" containerID="150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b"
Sep 17 23:39:09.753326 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.753236    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b"} err="failed to get container status \"150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b\": rpc error: code = NotFound desc = could not find container \"150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b\": container with ID starting with 150c9002a4d90e7949f1ff41040754b874145d960f2ddda8ddc0cf1fd260934b not found: ID does not exist"
Sep 17 23:39:09.753326 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.753262    2580 scope.go:117] "RemoveContainer" containerID="de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f"
Sep 17 23:39:09.753535 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:39:09.753518    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f\": container with ID starting with de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f not found: ID does not exist" containerID="de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f"
Sep 17 23:39:09.753623 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.753543    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f"} err="failed to get container status \"de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f\": rpc error: code = NotFound desc = could not find container \"de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f\": container with ID starting with de5e22f5d55c01789481b9766eb80d386a73768519dae0728126475d2c0f796f not found: ID does not exist"
Sep 17 23:39:09.754772 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.754754    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"]
Sep 17 23:39:09.761027 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:09.761006    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6275e9-openshift-default-79c5b8f78-w64l7"]
Sep 17 23:39:10.107961 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:10.107878    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" path="/var/lib/kubelet/pods/32b101bd-be25-4f15-b585-5f45a64b1dcd/volumes"
Sep 17 23:39:30.485580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:30.485485    2580 scope.go:117] "RemoveContainer" containerID="6fbad6055323a2d041397e68d0cb2b1a57eaf4464bff941a6a6793cd792bd5d3"
Sep 17 23:39:30.493938 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:30.493916    2580 scope.go:117] "RemoveContainer" containerID="284cfde79198c16ef8fa074d05279e4a9d8e17512768518d2cfc6db4f3f6d1c9"
Sep 17 23:39:38.710729 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:38.710696    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:38.810592 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:39:38.810563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:27.180078 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:40:27.180041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:28.842908 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:40:28.842869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:41:28.506698 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:28.506663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:41:28.605046 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:28.605011    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:41:30.204841 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:30.204793    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:41:30.207375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:30.207355    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rfdm4_d97c14bc-f869-4487-b479-4576d1dd4493/ovn-acl-logging/0.log"
Sep 17 23:41:43.686639 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:43.686600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:41:43.741284 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:43.741252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:41:56.358495 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:56.358456    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"]
Sep 17 23:41:56.358968 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:56.358696    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerName="manager" containerID="cri-o://5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158" gracePeriod=10
Sep 17 23:41:57.308761 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.308732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:41:57.385155 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.385124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad5da990-f291-48f6-91dd-96b54e91d5c3-extensions-socket-volume\") pod \"ad5da990-f291-48f6-91dd-96b54e91d5c3\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") "
Sep 17 23:41:57.385552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.385182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlmqz\" (UniqueName: \"kubernetes.io/projected/ad5da990-f291-48f6-91dd-96b54e91d5c3-kube-api-access-tlmqz\") pod \"ad5da990-f291-48f6-91dd-96b54e91d5c3\" (UID: \"ad5da990-f291-48f6-91dd-96b54e91d5c3\") "
Sep 17 23:41:57.385552 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.385468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5da990-f291-48f6-91dd-96b54e91d5c3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad5da990-f291-48f6-91dd-96b54e91d5c3" (UID: "ad5da990-f291-48f6-91dd-96b54e91d5c3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:41:57.387466 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.387443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5da990-f291-48f6-91dd-96b54e91d5c3-kube-api-access-tlmqz" (OuterVolumeSpecName: "kube-api-access-tlmqz") pod "ad5da990-f291-48f6-91dd-96b54e91d5c3" (UID: "ad5da990-f291-48f6-91dd-96b54e91d5c3"). InnerVolumeSpecName "kube-api-access-tlmqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:41:57.486639 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.486541    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad5da990-f291-48f6-91dd-96b54e91d5c3-extensions-socket-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:41:57.486639 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.486576    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmqz\" (UniqueName: \"kubernetes.io/projected/ad5da990-f291-48f6-91dd-96b54e91d5c3-kube-api-access-tlmqz\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:41:57.571516 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.571481    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerID="5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158" exitCode=0
Sep 17 23:41:57.571691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.571526    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"
Sep 17 23:41:57.571691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.571563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" event={"ID":"ad5da990-f291-48f6-91dd-96b54e91d5c3","Type":"ContainerDied","Data":"5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158"}
Sep 17 23:41:57.571691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.571619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f" event={"ID":"ad5da990-f291-48f6-91dd-96b54e91d5c3","Type":"ContainerDied","Data":"9257248d4701bdf1afe2062b7ee44fc16a2813ef8c2853d741c6feb8cd4447dd"}
Sep 17 23:41:57.571691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.571645    2580 scope.go:117] "RemoveContainer" containerID="5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158"
Sep 17 23:41:57.586560 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.586535    2580 scope.go:117] "RemoveContainer" containerID="3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92"
Sep 17 23:41:57.595366 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.595345    2580 scope.go:117] "RemoveContainer" containerID="5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158"
Sep 17 23:41:57.595630 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:41:57.595612    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158\": container with ID starting with 5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158 not found: ID does not exist" containerID="5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158"
Sep 17 23:41:57.595715 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.595637    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158"} err="failed to get container status \"5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158\": rpc error: code = NotFound desc = could not find container \"5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158\": container with ID starting with 5762e7ec1cdd267fba4031e42d17cdeb82134317b6ac736029d0df1608363158 not found: ID does not exist"
Sep 17 23:41:57.595715 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.595652    2580 scope.go:117] "RemoveContainer" containerID="3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92"
Sep 17 23:41:57.596038 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:41:57.595997    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92\": container with ID starting with 3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92 not found: ID does not exist" containerID="3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92"
Sep 17 23:41:57.596162 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.596042    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92"} err="failed to get container status \"3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92\": rpc error: code = NotFound desc = could not find container \"3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92\": container with ID starting with 3aee11eebb0c64d0dd243aef02d75e59133a35dfc5b50114146a889e0656ea92 not found: ID does not exist"
Sep 17 23:41:57.597303 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.597282    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"]
Sep 17 23:41:57.600289 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:57.600268    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kg65f"]
Sep 17 23:41:58.108875 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:41:58.108842    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" path="/var/lib/kubelet/pods/ad5da990-f291-48f6-91dd-96b54e91d5c3/volumes"
Sep 17 23:42:30.542568 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:30.542491    2580 scope.go:117] "RemoveContainer" containerID="04bb79f4f32ef3626bcd379217fcc650e259515e22d35905b49dc1f995d67ecd"
Sep 17 23:42:57.008990 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.008952    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"]
Sep 17 23:42:57.009375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009272    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerName="manager"
Sep 17 23:42:57.009375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009283    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerName="manager"
Sep 17 23:42:57.009375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009296    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a60ef7-d072-452a-b6c4-97022e2e94e4" containerName="revoke-keys"
Sep 17 23:42:57.009375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009301    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a60ef7-d072-452a-b6c4-97022e2e94e4" containerName="revoke-keys"
Sep 17 23:42:57.009375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009312    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy"
Sep 17 23:42:57.009375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009318    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy"
Sep 17 23:42:57.009375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009374    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerName="manager"
Sep 17 23:42:57.009597 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009382    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy"
Sep 17 23:42:57.009597 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009388    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4756fd94-7c01-40b2-96f3-098a0cce75dd" containerName="istio-proxy"
Sep 17 23:42:57.009597 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009394    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a60ef7-d072-452a-b6c4-97022e2e94e4" containerName="revoke-keys"
Sep 17 23:42:57.009597 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.009399    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerName="manager"
Sep 17 23:42:57.012274 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.012256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.014878 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.014845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t687b\""
Sep 17 23:42:57.062760 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.062721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"]
Sep 17 23:42:57.094686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.094656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/001378ea-c60c-4ece-b3d3-315a31041795-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7gcsw\" (UID: \"001378ea-c60c-4ece-b3d3-315a31041795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.094882 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.094768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pqjj\" (UniqueName: \"kubernetes.io/projected/001378ea-c60c-4ece-b3d3-315a31041795-kube-api-access-6pqjj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7gcsw\" (UID: \"001378ea-c60c-4ece-b3d3-315a31041795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.196316 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.196279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqjj\" (UniqueName: \"kubernetes.io/projected/001378ea-c60c-4ece-b3d3-315a31041795-kube-api-access-6pqjj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7gcsw\" (UID: \"001378ea-c60c-4ece-b3d3-315a31041795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.196517 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.196363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/001378ea-c60c-4ece-b3d3-315a31041795-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7gcsw\" (UID: \"001378ea-c60c-4ece-b3d3-315a31041795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.196750 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.196729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/001378ea-c60c-4ece-b3d3-315a31041795-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7gcsw\" (UID: \"001378ea-c60c-4ece-b3d3-315a31041795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.207669 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.207644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqjj\" (UniqueName: \"kubernetes.io/projected/001378ea-c60c-4ece-b3d3-315a31041795-kube-api-access-6pqjj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7gcsw\" (UID: \"001378ea-c60c-4ece-b3d3-315a31041795\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.323161 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.323049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.606526 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.606500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"]
Sep 17 23:42:57.609488 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:42:57.609459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001378ea_c60c_4ece_b3d3_315a31041795.slice/crio-e910947ef6e355c7904ed635b3fd5c0c18ec80c8423ceecb6a81a92221b72410 WatchSource:0}: Error finding container e910947ef6e355c7904ed635b3fd5c0c18ec80c8423ceecb6a81a92221b72410: Status 404 returned error can't find the container with id e910947ef6e355c7904ed635b3fd5c0c18ec80c8423ceecb6a81a92221b72410
Sep 17 23:42:57.864978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.864887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw" event={"ID":"001378ea-c60c-4ece-b3d3-315a31041795","Type":"ContainerStarted","Data":"e71dc7100c8eca3d5567930a46f832e22f14c5a42b448ba2f68c16a0392b25cf"}
Sep 17 23:42:57.864978 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.864947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw" event={"ID":"001378ea-c60c-4ece-b3d3-315a31041795","Type":"ContainerStarted","Data":"e910947ef6e355c7904ed635b3fd5c0c18ec80c8423ceecb6a81a92221b72410"}
Sep 17 23:42:57.865187 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.865040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:42:57.888316 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:42:57.888251    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw" podStartSLOduration=1.888231707 podStartE2EDuration="1.888231707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:42:57.88600625 +0000 UTC m=+2188.315866085" watchObservedRunningTime="2026-09-17 23:42:57.888231707 +0000 UTC m=+2188.318091523"
Sep 17 23:43:08.872534 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:08.872503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7gcsw"
Sep 17 23:43:30.564589 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:30.564554    2580 scope.go:117] "RemoveContainer" containerID="734db7efc395a3f5ac485b18fce995d7938dc07ce7c8bf9e3b3ba6ca35412e3b"
Sep 17 23:43:33.235243 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.235208    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"]
Sep 17 23:43:33.235614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.235559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy"
Sep 17 23:43:33.235614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.235571    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy"
Sep 17 23:43:33.235614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.235590    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerName="manager"
Sep 17 23:43:33.235614 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.235595    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5da990-f291-48f6-91dd-96b54e91d5c3" containerName="manager"
Sep 17 23:43:33.235826 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.235665    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b101bd-be25-4f15-b585-5f45a64b1dcd" containerName="istio-proxy"
Sep 17 23:43:33.240245 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.240224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.242613 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.242587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3b2541e9-kserve-self-signed-certs\""
Sep 17 23:43:33.247128 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.247100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"]
Sep 17 23:43:33.328295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.328243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfm7s\" (UniqueName: \"kubernetes.io/projected/6760a163-8c28-46fa-a854-08becaa24420-kube-api-access-cfm7s\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.328503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.328323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a163-8c28-46fa-a854-08becaa24420-tls-certs\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.328503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.328353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-home\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.328503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.328392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-tmp-dir\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.328503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.328416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-dshm\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.328503 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.328437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-model-cache\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.359685 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.359656    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:33.429181 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfm7s\" (UniqueName: \"kubernetes.io/projected/6760a163-8c28-46fa-a854-08becaa24420-kube-api-access-cfm7s\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429361 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a163-8c28-46fa-a854-08becaa24420-tls-certs\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429361 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-home\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429361 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-tmp-dir\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429361 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-dshm\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429361 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-model-cache\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429726 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-home\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429837 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-model-cache\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.429837 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.429781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-tmp-dir\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.431854 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.431828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-dshm\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.431969 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.431956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a163-8c28-46fa-a854-08becaa24420-tls-certs\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.437098 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.437076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfm7s\" (UniqueName: \"kubernetes.io/projected/6760a163-8c28-46fa-a854-08becaa24420-kube-api-access-cfm7s\") pod \"e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") " pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.550927 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.550825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:33.823008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:33.822928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"]
Sep 17 23:43:33.826034 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:43:33.826006    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6760a163_8c28_46fa_a854_08becaa24420.slice/crio-66000af00145ed07076ef1d7682dbdfd69676662c3db09d79cd1d7b83145021b WatchSource:0}: Error finding container 66000af00145ed07076ef1d7682dbdfd69676662c3db09d79cd1d7b83145021b: Status 404 returned error can't find the container with id 66000af00145ed07076ef1d7682dbdfd69676662c3db09d79cd1d7b83145021b
Sep 17 23:43:34.042534 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:34.042496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw" event={"ID":"6760a163-8c28-46fa-a854-08becaa24420","Type":"ContainerStarted","Data":"6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8"}
Sep 17 23:43:34.042706 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:34.042542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw" event={"ID":"6760a163-8c28-46fa-a854-08becaa24420","Type":"ContainerStarted","Data":"66000af00145ed07076ef1d7682dbdfd69676662c3db09d79cd1d7b83145021b"}
Sep 17 23:43:34.059781 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:34.059725    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw" podStartSLOduration=1.059709854 podStartE2EDuration="1.059709854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:43:34.059011692 +0000 UTC m=+2224.488871514" watchObservedRunningTime="2026-09-17 23:43:34.059709854 +0000 UTC m=+2224.489569672"
Sep 17 23:43:39.937567 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.937527    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"]
Sep 17 23:43:39.940686 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.940661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:39.942924 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.942899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3b2541e9-kserve-self-signed-certs\""
Sep 17 23:43:39.950007 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.949983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"]
Sep 17 23:43:39.961400 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.961374    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:39.993185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.993154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bb8d4d-126f-40ef-833b-9c9537444211-tls-certs\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:39.993185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.993198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-home\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:39.993415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.993223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-dshm\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:39.993415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.993358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-model-cache\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:39.993415 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.993409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-tmp-dir\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:39.993522 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:39.993429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shm7\" (UniqueName: \"kubernetes.io/projected/b2bb8d4d-126f-40ef-833b-9c9537444211-kube-api-access-6shm7\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.094816 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.094766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-dshm\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.094856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-model-cache\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.094899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-tmp-dir\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.094915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shm7\" (UniqueName: \"kubernetes.io/projected/b2bb8d4d-126f-40ef-833b-9c9537444211-kube-api-access-6shm7\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.094968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bb8d4d-126f-40ef-833b-9c9537444211-tls-certs\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.095007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-home\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095389 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.095368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-home\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095525 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.095400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-tmp-dir\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.095579 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.095550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-model-cache\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.097371 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.097340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-dshm\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.097763 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.097744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bb8d4d-126f-40ef-833b-9c9537444211-tls-certs\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.103065 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.103037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shm7\" (UniqueName: \"kubernetes.io/projected/b2bb8d4d-126f-40ef-833b-9c9537444211-kube-api-access-6shm7\") pod \"e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") " pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.235043 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.234947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:40.250953 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.250917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:40.525582 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:40.525556    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"]
Sep 17 23:43:40.527868 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:43:40.527835    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2bb8d4d_126f_40ef_833b_9c9537444211.slice/crio-0faf69eeaeba7ee57954a9fd228b578e71cfd5bfdc9c2a83d81df3e2ea0c30df WatchSource:0}: Error finding container 0faf69eeaeba7ee57954a9fd228b578e71cfd5bfdc9c2a83d81df3e2ea0c30df: Status 404 returned error can't find the container with id 0faf69eeaeba7ee57954a9fd228b578e71cfd5bfdc9c2a83d81df3e2ea0c30df
Sep 17 23:43:41.084941 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:41.084891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7" event={"ID":"b2bb8d4d-126f-40ef-833b-9c9537444211","Type":"ContainerStarted","Data":"6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd"}
Sep 17 23:43:41.084941 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:41.084948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7" event={"ID":"b2bb8d4d-126f-40ef-833b-9c9537444211","Type":"ContainerStarted","Data":"0faf69eeaeba7ee57954a9fd228b578e71cfd5bfdc9c2a83d81df3e2ea0c30df"}
Sep 17 23:43:41.101183 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:41.101090    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7" podStartSLOduration=2.101066617 podStartE2EDuration="2.101066617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:43:41.100550801 +0000 UTC m=+2231.530410620" watchObservedRunningTime="2026-09-17 23:43:41.101066617 +0000 UTC m=+2231.530926436"
Sep 17 23:43:43.551583 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:43.551545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:43.552003 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:43.551599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:43.564204 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:43.564180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:44.112100 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:44.112071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:43:47.810125 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:47.810071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:50.251576 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:50.251536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:50.252092 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:50.251592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:50.264506 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:50.264478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:51.144819 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:51.144766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:43:52.280967 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:52.280922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:52.337542 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:43:52.337510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:04.401202 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:04.401168    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"]
Sep 17 23:44:04.421299 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:04.421257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:05.455097 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:05.455065    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"]
Sep 17 23:44:05.468765 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:05.468736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:07.207106 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:07.207061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:19.421491 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.421444    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw" podUID="6760a163-8c28-46fa-a854-08becaa24420" containerName="main" containerID="cri-o://6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8" gracePeriod=45
Sep 17 23:44:19.673122 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.673058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:44:19.773844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.773789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a163-8c28-46fa-a854-08becaa24420-tls-certs\") pod \"6760a163-8c28-46fa-a854-08becaa24420\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") "
Sep 17 23:44:19.773844 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.773850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfm7s\" (UniqueName: \"kubernetes.io/projected/6760a163-8c28-46fa-a854-08becaa24420-kube-api-access-cfm7s\") pod \"6760a163-8c28-46fa-a854-08becaa24420\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") "
Sep 17 23:44:19.774110 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.773937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-dshm\") pod \"6760a163-8c28-46fa-a854-08becaa24420\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") "
Sep 17 23:44:19.774110 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.773960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-home\") pod \"6760a163-8c28-46fa-a854-08becaa24420\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") "
Sep 17 23:44:19.774110 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.773976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-model-cache\") pod \"6760a163-8c28-46fa-a854-08becaa24420\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") "
Sep 17 23:44:19.774110 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.774025    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-tmp-dir\") pod \"6760a163-8c28-46fa-a854-08becaa24420\" (UID: \"6760a163-8c28-46fa-a854-08becaa24420\") "
Sep 17 23:44:19.774314 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.774276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-home" (OuterVolumeSpecName: "home") pod "6760a163-8c28-46fa-a854-08becaa24420" (UID: "6760a163-8c28-46fa-a854-08becaa24420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:19.774314 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.774292    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-model-cache" (OuterVolumeSpecName: "model-cache") pod "6760a163-8c28-46fa-a854-08becaa24420" (UID: "6760a163-8c28-46fa-a854-08becaa24420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:19.774419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.774389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6760a163-8c28-46fa-a854-08becaa24420" (UID: "6760a163-8c28-46fa-a854-08becaa24420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:19.776384 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.776353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-dshm" (OuterVolumeSpecName: "dshm") pod "6760a163-8c28-46fa-a854-08becaa24420" (UID: "6760a163-8c28-46fa-a854-08becaa24420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:19.776474 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.776416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6760a163-8c28-46fa-a854-08becaa24420-kube-api-access-cfm7s" (OuterVolumeSpecName: "kube-api-access-cfm7s") pod "6760a163-8c28-46fa-a854-08becaa24420" (UID: "6760a163-8c28-46fa-a854-08becaa24420"). InnerVolumeSpecName "kube-api-access-cfm7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:19.776474 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.776422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6760a163-8c28-46fa-a854-08becaa24420-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6760a163-8c28-46fa-a854-08becaa24420" (UID: "6760a163-8c28-46fa-a854-08becaa24420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:19.875419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.875379    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6760a163-8c28-46fa-a854-08becaa24420-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:19.875419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.875412    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfm7s\" (UniqueName: \"kubernetes.io/projected/6760a163-8c28-46fa-a854-08becaa24420-kube-api-access-cfm7s\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:19.875419 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.875425    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:19.875667 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.875436    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:19.875667 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.875450    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:19.875667 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:19.875463    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6760a163-8c28-46fa-a854-08becaa24420-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:20.279929 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.279892    2580 generic.go:356] "Generic (PLEG): container finished" podID="6760a163-8c28-46fa-a854-08becaa24420" containerID="6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8" exitCode=0
Sep 17 23:44:20.280114 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.279934    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"
Sep 17 23:44:20.280114 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.279951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw" event={"ID":"6760a163-8c28-46fa-a854-08becaa24420","Type":"ContainerDied","Data":"6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8"}
Sep 17 23:44:20.280114 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.279997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw" event={"ID":"6760a163-8c28-46fa-a854-08becaa24420","Type":"ContainerDied","Data":"66000af00145ed07076ef1d7682dbdfd69676662c3db09d79cd1d7b83145021b"}
Sep 17 23:44:20.280114 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.280017    2580 scope.go:117] "RemoveContainer" containerID="6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8"
Sep 17 23:44:20.289500 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.289484    2580 scope.go:117] "RemoveContainer" containerID="6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8"
Sep 17 23:44:20.289794 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:44:20.289776    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8\": container with ID starting with 6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8 not found: ID does not exist" containerID="6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8"
Sep 17 23:44:20.289876 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.289816    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8"} err="failed to get container status \"6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8\": rpc error: code = NotFound desc = could not find container \"6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8\": container with ID starting with 6999bc24c7e418600be2c64ae419286cd8f2df8b2bb9217c6cdd31a2ce3bb0a8 not found: ID does not exist"
Sep 17 23:44:20.295594 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.295572    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"]
Sep 17 23:44:20.298943 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.298920    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3b2541e9-kserve-7f767cbf95-xtblw"]
Sep 17 23:44:20.476824 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.476758    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7" podUID="b2bb8d4d-126f-40ef-833b-9c9537444211" containerName="main" containerID="cri-o://6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd" gracePeriod=45
Sep 17 23:44:20.746241 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.746215    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:44:20.782468 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.782434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-dshm\") pod \"b2bb8d4d-126f-40ef-833b-9c9537444211\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") "
Sep 17 23:44:20.782644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.782555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bb8d4d-126f-40ef-833b-9c9537444211-tls-certs\") pod \"b2bb8d4d-126f-40ef-833b-9c9537444211\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") "
Sep 17 23:44:20.782644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.782585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-model-cache\") pod \"b2bb8d4d-126f-40ef-833b-9c9537444211\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") "
Sep 17 23:44:20.782644 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.782617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6shm7\" (UniqueName: \"kubernetes.io/projected/b2bb8d4d-126f-40ef-833b-9c9537444211-kube-api-access-6shm7\") pod \"b2bb8d4d-126f-40ef-833b-9c9537444211\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") "
Sep 17 23:44:20.782786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.782645    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-home\") pod \"b2bb8d4d-126f-40ef-833b-9c9537444211\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") "
Sep 17 23:44:20.782786 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.782695    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-tmp-dir\") pod \"b2bb8d4d-126f-40ef-833b-9c9537444211\" (UID: \"b2bb8d4d-126f-40ef-833b-9c9537444211\") "
Sep 17 23:44:20.783270 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.783240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2bb8d4d-126f-40ef-833b-9c9537444211" (UID: "b2bb8d4d-126f-40ef-833b-9c9537444211"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:20.783406 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.783345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-home" (OuterVolumeSpecName: "home") pod "b2bb8d4d-126f-40ef-833b-9c9537444211" (UID: "b2bb8d4d-126f-40ef-833b-9c9537444211"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:20.783406 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.783361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2bb8d4d-126f-40ef-833b-9c9537444211" (UID: "b2bb8d4d-126f-40ef-833b-9c9537444211"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:20.785252 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.785226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-dshm" (OuterVolumeSpecName: "dshm") pod "b2bb8d4d-126f-40ef-833b-9c9537444211" (UID: "b2bb8d4d-126f-40ef-833b-9c9537444211"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:20.785703 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.785673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bb8d4d-126f-40ef-833b-9c9537444211-kube-api-access-6shm7" (OuterVolumeSpecName: "kube-api-access-6shm7") pod "b2bb8d4d-126f-40ef-833b-9c9537444211" (UID: "b2bb8d4d-126f-40ef-833b-9c9537444211"). InnerVolumeSpecName "kube-api-access-6shm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:20.785703 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.785677    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bb8d4d-126f-40ef-833b-9c9537444211-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2bb8d4d-126f-40ef-833b-9c9537444211" (UID: "b2bb8d4d-126f-40ef-833b-9c9537444211"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:20.884154 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.884108    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-home\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:20.884154 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.884163    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-tmp-dir\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:20.884423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.884173    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-dshm\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:20.884423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.884180    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bb8d4d-126f-40ef-833b-9c9537444211-tls-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:20.884423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.884189    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bb8d4d-126f-40ef-833b-9c9537444211-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:20.884423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:20.884197    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shm7\" (UniqueName: \"kubernetes.io/projected/b2bb8d4d-126f-40ef-833b-9c9537444211-kube-api-access-6shm7\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:21.288253 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.288220    2580 generic.go:356] "Generic (PLEG): container finished" podID="b2bb8d4d-126f-40ef-833b-9c9537444211" containerID="6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd" exitCode=0
Sep 17 23:44:21.288483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.288262    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"
Sep 17 23:44:21.288483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.288278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7" event={"ID":"b2bb8d4d-126f-40ef-833b-9c9537444211","Type":"ContainerDied","Data":"6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd"}
Sep 17 23:44:21.288483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.288306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7" event={"ID":"b2bb8d4d-126f-40ef-833b-9c9537444211","Type":"ContainerDied","Data":"0faf69eeaeba7ee57954a9fd228b578e71cfd5bfdc9c2a83d81df3e2ea0c30df"}
Sep 17 23:44:21.288483 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.288324    2580 scope.go:117] "RemoveContainer" containerID="6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd"
Sep 17 23:44:21.297480 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.297459    2580 scope.go:117] "RemoveContainer" containerID="6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd"
Sep 17 23:44:21.297765 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:44:21.297745    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd\": container with ID starting with 6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd not found: ID does not exist" containerID="6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd"
Sep 17 23:44:21.297824 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.297776    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd"} err="failed to get container status \"6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd\": rpc error: code = NotFound desc = could not find container \"6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd\": container with ID starting with 6c5a107dc3532f37df69246e6a69fcdc482eef2bc4d426d01418bb8e82e6e3cd not found: ID does not exist"
Sep 17 23:44:21.308626 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.308602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"]
Sep 17 23:44:21.311905 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:21.311882    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3b2541e9-kserve-77768557f-bzzl7"]
Sep 17 23:44:22.114497 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:22.114459    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6760a163-8c28-46fa-a854-08becaa24420" path="/var/lib/kubelet/pods/6760a163-8c28-46fa-a854-08becaa24420/volumes"
Sep 17 23:44:22.115005 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:22.114878    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2bb8d4d-126f-40ef-833b-9c9537444211" path="/var/lib/kubelet/pods/b2bb8d4d-126f-40ef-833b-9c9537444211/volumes"
Sep 17 23:44:24.419339 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.419297    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"]
Sep 17 23:44:24.419855 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.419838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2bb8d4d-126f-40ef-833b-9c9537444211" containerName="main"
Sep 17 23:44:24.419946 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.419859    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2bb8d4d-126f-40ef-833b-9c9537444211" containerName="main"
Sep 17 23:44:24.419946 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.419877    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6760a163-8c28-46fa-a854-08becaa24420" containerName="main"
Sep 17 23:44:24.419946 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.419885    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6760a163-8c28-46fa-a854-08becaa24420" containerName="main"
Sep 17 23:44:24.420101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.419974    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2bb8d4d-126f-40ef-833b-9c9537444211" containerName="main"
Sep 17 23:44:24.420101 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.419988    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6760a163-8c28-46fa-a854-08becaa24420" containerName="main"
Sep 17 23:44:24.424370 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.424347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.427578 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.427553    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:44:24.427731 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.427599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 23:44:24.427731 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.427610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 23:44:24.427731 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.427705    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 23:44:24.427937 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.427753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xltjn\""
Sep 17 23:44:24.435737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.435715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"]
Sep 17 23:44:24.519145 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.519108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-maas-api-tls\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.519321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.519156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-openshift-service-ca\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.519321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.519188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-metrics-certs\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.519321 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.519206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25kv\" (UniqueName: \"kubernetes.io/projected/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-kube-api-access-b25kv\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.620500 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.620458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-maas-api-tls\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.620691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.620511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-openshift-service-ca\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.620691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.620546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-metrics-certs\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.620691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.620562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25kv\" (UniqueName: \"kubernetes.io/projected/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-kube-api-access-b25kv\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.621439 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.621407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-openshift-service-ca\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.623215 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.623195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-metrics-certs\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.623292 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.623242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-maas-api-tls\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.628341 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.628318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25kv\" (UniqueName: \"kubernetes.io/projected/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-kube-api-access-b25kv\") pod \"maas-api-7bb7f9d9f-dxfq2\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") " pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:24.735346 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:24.735253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:25.006997 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:25.006915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"]
Sep 17 23:44:25.009770 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:44:25.009740    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd1ba58_9804_47f4_9ec0_b308ce2c19cd.slice/crio-510e8ed38c19631c5bd291330e660d63f024cf72407fd81bd85be24233de547f WatchSource:0}: Error finding container 510e8ed38c19631c5bd291330e660d63f024cf72407fd81bd85be24233de547f: Status 404 returned error can't find the container with id 510e8ed38c19631c5bd291330e660d63f024cf72407fd81bd85be24233de547f
Sep 17 23:44:25.011140 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:25.011119    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:44:25.310759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:25.310676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2" event={"ID":"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd","Type":"ContainerStarted","Data":"510e8ed38c19631c5bd291330e660d63f024cf72407fd81bd85be24233de547f"}
Sep 17 23:44:28.330361 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:28.330324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2" event={"ID":"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd","Type":"ContainerStarted","Data":"3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93"}
Sep 17 23:44:28.330771 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:28.330400    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:28.348079 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:28.348031    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2" podStartSLOduration=1.943545239 podStartE2EDuration="4.348017679s" totalImagesPullingTime="2.40447244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:28.34658234 +0000 UTC m=+2278.776442158" watchObservedRunningTime="2026-09-17 23:44:28.348017679 +0000 UTC m=+2278.777877497"
Sep 17 23:44:30.587486 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:30.587443    2580 scope.go:117] "RemoveContainer" containerID="d40c01cc5fca49b4764ee66269af4422c1ffb09cda3c3408bf1632e564baf6a2"
Sep 17 23:44:30.596421 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:30.596399    2580 scope.go:117] "RemoveContainer" containerID="2322f05203954c670f8da1d1fa585c336e6343fc97afb1d2d0042c82f40968df"
Sep 17 23:44:31.761269 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.761235    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"]
Sep 17 23:44:31.761689 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.761409    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2" podUID="5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" containerName="maas-api" containerID="cri-o://3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93" gracePeriod=30
Sep 17 23:44:31.793276 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.793243    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"]
Sep 17 23:44:31.796757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.796735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.807358 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.807318    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"]
Sep 17 23:44:31.892564 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.892526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-metrics-certs\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.892712 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.892588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7559b\" (UniqueName: \"kubernetes.io/projected/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-kube-api-access-7559b\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.892712 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.892664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-maas-api-tls\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.892712 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.892685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-openshift-service-ca\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.994008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.993952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-maas-api-tls\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.994008 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.994004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-openshift-service-ca\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.994185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.994097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-metrics-certs\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.994185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.994146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7559b\" (UniqueName: \"kubernetes.io/projected/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-kube-api-access-7559b\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.995081 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.995054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-openshift-service-ca\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.996685 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.996664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-maas-api-tls\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:31.997319 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:31.997301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-metrics-certs\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:32.001599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.001574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7559b\" (UniqueName: \"kubernetes.io/projected/a6f75639-4feb-4e45-99f9-6d71fa5a28aa-kube-api-access-7559b\") pod \"maas-api-d7bf8bdb7-bhn2j\" (UID: \"a6f75639-4feb-4e45-99f9-6d71fa5a28aa\") " pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:32.019904 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.019849    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:32.094977 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.094935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-openshift-service-ca\") pod \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") "
Sep 17 23:44:32.095185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.094986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-maas-api-tls\") pod \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") "
Sep 17 23:44:32.095185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.095047    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b25kv\" (UniqueName: \"kubernetes.io/projected/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-kube-api-access-b25kv\") pod \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") "
Sep 17 23:44:32.095185 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.095144    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-metrics-certs\") pod \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\" (UID: \"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd\") "
Sep 17 23:44:32.095433 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.095402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" (UID: "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:32.095601 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.095579    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-openshift-service-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:32.097498 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.097465    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" (UID: "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:32.097596 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.097501    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" (UID: "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:32.097596 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.097562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-kube-api-access-b25kv" (OuterVolumeSpecName: "kube-api-access-b25kv") pod "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" (UID: "5dd1ba58-9804-47f4-9ec0-b308ce2c19cd"). InnerVolumeSpecName "kube-api-access-b25kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:32.146354 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.146321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:32.197229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.197156    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25kv\" (UniqueName: \"kubernetes.io/projected/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-kube-api-access-b25kv\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:32.197229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.197192    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-metrics-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:32.197229 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.197207    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd-maas-api-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
Sep 17 23:44:32.351641 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.351562    2580 generic.go:356] "Generic (PLEG): container finished" podID="5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" containerID="3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93" exitCode=0
Sep 17 23:44:32.351641 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.351602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2" event={"ID":"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd","Type":"ContainerDied","Data":"3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93"}
Sep 17 23:44:32.351641 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.351610    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"
Sep 17 23:44:32.351641 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.351642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2" event={"ID":"5dd1ba58-9804-47f4-9ec0-b308ce2c19cd","Type":"ContainerDied","Data":"510e8ed38c19631c5bd291330e660d63f024cf72407fd81bd85be24233de547f"}
Sep 17 23:44:32.352002 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.351663    2580 scope.go:117] "RemoveContainer" containerID="3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93"
Sep 17 23:44:32.360348 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.360331    2580 scope.go:117] "RemoveContainer" containerID="3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93"
Sep 17 23:44:32.360635 ip-10-0-139-248 kubenswrapper[2580]: E0917 23:44:32.360615    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93\": container with ID starting with 3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93 not found: ID does not exist" containerID="3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93"
Sep 17 23:44:32.360683 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.360646    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93"} err="failed to get container status \"3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93\": rpc error: code = NotFound desc = could not find container \"3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93\": container with ID starting with 3c354d9442496b5e84ab2cd51fd523150230440d96ab815198b7abb9164f7f93 not found: ID does not exist"
Sep 17 23:44:32.367236 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.367208    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"]
Sep 17 23:44:32.370787 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.370762    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bb7f9d9f-dxfq2"]
Sep 17 23:44:32.418636 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:32.418603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"]
Sep 17 23:44:32.420696 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:44:32.420668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f75639_4feb_4e45_99f9_6d71fa5a28aa.slice/crio-9b843a387c0538994d4bd35d355da55d958fe9d7ea2544849ad52dbe303ae525 WatchSource:0}: Error finding container 9b843a387c0538994d4bd35d355da55d958fe9d7ea2544849ad52dbe303ae525: Status 404 returned error can't find the container with id 9b843a387c0538994d4bd35d355da55d958fe9d7ea2544849ad52dbe303ae525
Sep 17 23:44:33.358599 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:33.358563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j" event={"ID":"a6f75639-4feb-4e45-99f9-6d71fa5a28aa","Type":"ContainerStarted","Data":"f8f6aaafc857596091fcbca1555a5e17727c812ae44a142a4fd530b2522286b0"}
Sep 17 23:44:33.359010 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:33.358610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j" event={"ID":"a6f75639-4feb-4e45-99f9-6d71fa5a28aa","Type":"ContainerStarted","Data":"9b843a387c0538994d4bd35d355da55d958fe9d7ea2544849ad52dbe303ae525"}
Sep 17 23:44:33.359010 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:33.358687    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:33.379793 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:33.379743    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j" podStartSLOduration=2.005211121 podStartE2EDuration="2.379726247s" totalImagesPullingTime="374.515126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:33.378566903 +0000 UTC m=+2283.808426721" watchObservedRunningTime="2026-09-17 23:44:33.379726247 +0000 UTC m=+2283.809586084"
Sep 17 23:44:34.107630 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:34.107596    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" path="/var/lib/kubelet/pods/5dd1ba58-9804-47f4-9ec0-b308ce2c19cd/volumes"
Sep 17 23:44:39.369144 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:39.369115    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d7bf8bdb7-bhn2j"
Sep 17 23:44:45.442196 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:45.442162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:45.498756 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:44:45.498727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:00.197602 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:00.197571    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5976b594-fjzqw_890439e4-9ff6-4ade-983d-8064e83e606a/copy-manifests/0.log"
Sep 17 23:45:00.204295 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:00.204272    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5976b594-fjzqw_890439e4-9ff6-4ade-983d-8064e83e606a/manager/0.log"
Sep 17 23:45:00.684580 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:00.684551    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d7bf8bdb7-bhn2j_a6f75639-4feb-4e45-99f9-6d71fa5a28aa/maas-api/0.log"
Sep 17 23:45:02.108179 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:02.108146    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g6rsg_e7a87d72-1d6a-4278-a0c2-51f3b35e809a/manager/0.log"
Sep 17 23:45:02.215708 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:02.215679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-lhr6j_27571f8a-284d-47fc-9392-2278a95fa8f4/kuadrant-console-plugin/0.log"
Sep 17 23:45:02.335728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:02.335699    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7gcsw_001378ea-c60c-4ece-b3d3-315a31041795/manager/0.log"
Sep 17 23:45:03.154533 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:03.154502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8d65647-7mdqg_80dc2e8e-b2de-4d56-bed1-1b6f20e6021a/kube-auth-proxy/0.log"
Sep 17 23:45:03.493142 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:03.493046    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9jm5_cad23c97-6ac2-4db6-b447-2adaa97042e1/payload-pre-processing/0.log"
Sep 17 23:45:04.268395 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:04.268362    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8czwj_994ac596-c94b-420c-8692-29205ba0d9cd/main/0.log"
Sep 17 23:45:04.599633 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:04.599532    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2slcf_e640ea20-7cc8-4f38-b7b0-85502deee9ab/main/0.log"
Sep 17 23:45:04.707413 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:04.707381    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xw4kx_e11f00a4-8878-4361-9314-f20e96f4b0ff/main/0.log"
Sep 17 23:45:09.889242 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:09.889204    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:09.989331 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:09.989299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:12.903942 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:12.903905    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zlv7c_26053f12-63eb-48b5-a610-95ce5cd2c941/global-pull-secret-syncer/0.log"
Sep 17 23:45:12.951375 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:12.951342    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-498dh_217ae11d-95f6-4818-831d-700279f0fd51/konnectivity-agent/0.log"
Sep 17 23:45:13.087884 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:13.087851    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-248.ec2.internal_21013831982d43c783952b1c4eba40a7/haproxy/0.log"
Sep 17 23:45:17.602237 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:17.602127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g6rsg_e7a87d72-1d6a-4278-a0c2-51f3b35e809a/manager/0.log"
Sep 17 23:45:17.624757 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:17.624730    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-lhr6j_27571f8a-284d-47fc-9392-2278a95fa8f4/kuadrant-console-plugin/0.log"
Sep 17 23:45:17.684492 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:17.684461    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7gcsw_001378ea-c60c-4ece-b3d3-315a31041795/manager/0.log"
Sep 17 23:45:19.316737 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.316711    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6f4ms_edd19b0b-6819-4558-92f5-089bd1b674ef/kube-state-metrics/0.log"
Sep 17 23:45:19.341054 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.341010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6f4ms_edd19b0b-6819-4558-92f5-089bd1b674ef/kube-rbac-proxy-main/0.log"
Sep 17 23:45:19.362550 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.362528    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-6f4ms_edd19b0b-6819-4558-92f5-089bd1b674ef/kube-rbac-proxy-self/0.log"
Sep 17 23:45:19.388405 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.388360    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-678485c766-5b9fk_a9b7543c-2bc8-46b8-af17-56d1e6be1294/metrics-server/0.log"
Sep 17 23:45:19.412423 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.412344    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-vkbhw_a9b66e0b-4b3a-4c05-8536-37bb98f0c9ef/monitoring-plugin/0.log"
Sep 17 23:45:19.599412 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.599383    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rbtkm_8fc37975-cb27-4d09-9c60-916d3a88a503/node-exporter/0.log"
Sep 17 23:45:19.621598 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.621575    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rbtkm_8fc37975-cb27-4d09-9c60-916d3a88a503/kube-rbac-proxy/0.log"
Sep 17 23:45:19.643670 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.643591    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rbtkm_8fc37975-cb27-4d09-9c60-916d3a88a503/init-textfile/0.log"
Sep 17 23:45:19.677963 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.677895    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2bqjt_c383065b-e43f-4c7b-aa91-1a1845fcd488/kube-rbac-proxy-main/0.log"
Sep 17 23:45:19.708383 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.708355    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2bqjt_c383065b-e43f-4c7b-aa91-1a1845fcd488/kube-rbac-proxy-self/0.log"
Sep 17 23:45:19.731795 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.731767    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2bqjt_c383065b-e43f-4c7b-aa91-1a1845fcd488/openshift-state-metrics/0.log"
Sep 17 23:45:19.812743 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.812716    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25013661-efb7-4580-9b68-e00d37f0f5e4/prometheus/0.log"
Sep 17 23:45:19.839114 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.839080    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25013661-efb7-4580-9b68-e00d37f0f5e4/config-reloader/0.log"
Sep 17 23:45:19.868860 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.868828    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25013661-efb7-4580-9b68-e00d37f0f5e4/thanos-sidecar/0.log"
Sep 17 23:45:19.889976 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.889948    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25013661-efb7-4580-9b68-e00d37f0f5e4/kube-rbac-proxy-web/0.log"
Sep 17 23:45:19.915119 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.915094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25013661-efb7-4580-9b68-e00d37f0f5e4/kube-rbac-proxy/0.log"
Sep 17 23:45:19.935233 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.935165    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25013661-efb7-4580-9b68-e00d37f0f5e4/kube-rbac-proxy-thanos/0.log"
Sep 17 23:45:19.957010 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.956987    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25013661-efb7-4580-9b68-e00d37f0f5e4/init-config-reloader/0.log"
Sep 17 23:45:19.992294 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:19.992263    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-ppbfj_429aa7db-1cda-455d-844a-0e0a1e30b524/prometheus-operator/0.log"
Sep 17 23:45:20.011732 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:20.011704    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-ppbfj_429aa7db-1cda-455d-844a-0e0a1e30b524/kube-rbac-proxy/0.log"
Sep 17 23:45:22.822368 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.822273    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"]
Sep 17 23:45:22.822896 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.822876    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" containerName="maas-api"
Sep 17 23:45:22.823018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.822898    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" containerName="maas-api"
Sep 17 23:45:22.823018 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.823006    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd1ba58-9804-47f4-9ec0-b308ce2c19cd" containerName="maas-api"
Sep 17 23:45:22.826275 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.826253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:22.828618 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.828591    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xj26g\"/\"kube-root-ca.crt\""
Sep 17 23:45:22.828728 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.828686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xj26g\"/\"default-dockercfg-qmmhv\""
Sep 17 23:45:22.829484 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.829463    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xj26g\"/\"openshift-service-ca.crt\""
Sep 17 23:45:22.836809 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.836773    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"]
Sep 17 23:45:22.922122 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.922095    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-644f54f8c5-57q6p_66385567-6f11-43b7-b2a4-a112aa8edd7f/console/0.log"
Sep 17 23:45:22.958729 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.958690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-podres\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:22.958994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.958740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-lib-modules\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:22.958994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.958828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-sys\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:22.958994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.958861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghld\" (UniqueName: \"kubernetes.io/projected/2a1e4299-c452-4975-9413-5932bc94fdcc-kube-api-access-fghld\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:22.958994 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:22.958889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-proc\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060141 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-podres\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060141 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-lib-modules\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-sys\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghld\" (UniqueName: \"kubernetes.io/projected/2a1e4299-c452-4975-9413-5932bc94fdcc-kube-api-access-fghld\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-podres\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-sys\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-proc\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-proc\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.060393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.060321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1e4299-c452-4975-9413-5932bc94fdcc-lib-modules\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.069051 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.069020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghld\" (UniqueName: \"kubernetes.io/projected/2a1e4299-c452-4975-9413-5932bc94fdcc-kube-api-access-fghld\") pod \"perf-node-gather-daemonset-vl7d4\" (UID: \"2a1e4299-c452-4975-9413-5932bc94fdcc\") " pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.136424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.136341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.407449 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.407418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"]
Sep 17 23:45:23.411036 ip-10-0-139-248 kubenswrapper[2580]: W0917 23:45:23.411007    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a1e4299_c452_4975_9413_5932bc94fdcc.slice/crio-fcf1b758c45569a43a37edaa58aa315a52f8a6bcc5fb69e2999a979e8a4550da WatchSource:0}: Error finding container fcf1b758c45569a43a37edaa58aa315a52f8a6bcc5fb69e2999a979e8a4550da: Status 404 returned error can't find the container with id fcf1b758c45569a43a37edaa58aa315a52f8a6bcc5fb69e2999a979e8a4550da
Sep 17 23:45:23.607236 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.607195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4" event={"ID":"2a1e4299-c452-4975-9413-5932bc94fdcc","Type":"ContainerStarted","Data":"b0e482e6fb44353db6ccd8abb9d5e497bc6a202e66aa1e1ca1ea45e0b9051fd4"}
Sep 17 23:45:23.607429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.607258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4" event={"ID":"2a1e4299-c452-4975-9413-5932bc94fdcc","Type":"ContainerStarted","Data":"fcf1b758c45569a43a37edaa58aa315a52f8a6bcc5fb69e2999a979e8a4550da"}
Sep 17 23:45:23.607429 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.607282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:23.623424 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:23.623373    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4" podStartSLOduration=1.623356979 podStartE2EDuration="1.623356979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:23.621959179 +0000 UTC m=+2334.051819010" watchObservedRunningTime="2026-09-17 23:45:23.623356979 +0000 UTC m=+2334.053216814"
Sep 17 23:45:24.855493 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:24.855467    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sz9q8_b7d58559-fe6c-4175-9e0d-664079115ee3/dns/0.log"
Sep 17 23:45:24.874528 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:24.874500    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sz9q8_b7d58559-fe6c-4175-9e0d-664079115ee3/kube-rbac-proxy/0.log"
Sep 17 23:45:24.918662 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:24.918628    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6rzcj_f2fc1ca6-8c87-471a-bc2b-bb5f9fbb4d62/dns-node-resolver/0.log"
Sep 17 23:45:25.548251 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:25.548219    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hv4vh_8896e89b-3c26-4006-95f0-5620d676cb48/node-ca/0.log"
Sep 17 23:45:27.259288 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:27.259261    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8d65647-7mdqg_80dc2e8e-b2de-4d56-bed1-1b6f20e6021a/kube-auth-proxy/0.log"
Sep 17 23:45:27.361017 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:27.360989    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9jm5_cad23c97-6ac2-4db6-b447-2adaa97042e1/payload-pre-processing/0.log"
Sep 17 23:45:28.294181 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:28.294155    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bt4qj_c45bc469-07af-45ce-bfb0-aa730901da10/serve-healthcheck-canary/0.log"
Sep 17 23:45:28.912315 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:28.912283    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xp5xd_664bd601-ed30-4dab-9ba5-131d4b86e869/kube-rbac-proxy/0.log"
Sep 17 23:45:28.932695 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:28.932671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xp5xd_664bd601-ed30-4dab-9ba5-131d4b86e869/exporter/0.log"
Sep 17 23:45:28.955987 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:28.955962    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xp5xd_664bd601-ed30-4dab-9ba5-131d4b86e869/extractor/0.log"
Sep 17 23:45:29.622037 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:29.622005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xj26g/perf-node-gather-daemonset-vl7d4"
Sep 17 23:45:30.626232 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:30.626192    2580 scope.go:117] "RemoveContainer" containerID="8347bcdc390851ae2cda6a56d2b64f052a619bcf4b57cfb48a4d13716dd3205d"
Sep 17 23:45:31.814630 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:31.814597    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5976b594-fjzqw_890439e4-9ff6-4ade-983d-8064e83e606a/manager/0.log"
Sep 17 23:45:31.833788 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:31.833761    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5976b594-fjzqw_890439e4-9ff6-4ade-983d-8064e83e606a/copy-manifests/0.log"
Sep 17 23:45:33.181010 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:33.180979    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ccddb67c-xtbq4_0b34226a-924d-4cb7-a0d4-ada2a4a5b79a/manager/0.log"
Sep 17 23:45:33.208768 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:33.208743    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jdfjf_de8420c2-239b-4dcf-9b19-7ff5b48fe380/openshift-lws-operator/0.log"
Sep 17 23:45:33.455463 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:33.455371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8d65647-7mdqg_80dc2e8e-b2de-4d56-bed1-1b6f20e6021a/kube-auth-proxy/0.log"
Sep 17 23:45:33.523691 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:33.523659    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9jm5_cad23c97-6ac2-4db6-b447-2adaa97042e1/payload-pre-processing/0.log"
Sep 17 23:45:35.921156 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:35.921125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55b8d65647-7mdqg_80dc2e8e-b2de-4d56-bed1-1b6f20e6021a/kube-auth-proxy/0.log"
Sep 17 23:45:35.996393 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:35.996316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x9jm5_cad23c97-6ac2-4db6-b447-2adaa97042e1/payload-pre-processing/0.log"
Sep 17 23:45:39.826729 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:39.826701    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzs4g_cefaeeee-fef2-4a38-a4a1-d26e2b452228/kube-multus-additional-cni-plugins/0.log"
Sep 17 23:45:39.848827 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:39.848766    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzs4g_cefaeeee-fef2-4a38-a4a1-d26e2b452228/egress-router-binary-copy/0.log"
Sep 17 23:45:39.872759 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:39.872740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzs4g_cefaeeee-fef2-4a38-a4a1-d26e2b452228/cni-plugins/0.log"
Sep 17 23:45:39.898829 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:39.898790    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzs4g_cefaeeee-fef2-4a38-a4a1-d26e2b452228/bond-cni-plugin/0.log"
Sep 17 23:45:39.918539 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:39.918502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzs4g_cefaeeee-fef2-4a38-a4a1-d26e2b452228/routeoverride-cni/0.log"
Sep 17 23:45:39.937630 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:39.937592    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzs4g_cefaeeee-fef2-4a38-a4a1-d26e2b452228/whereabouts-cni-bincopy/0.log"
Sep 17 23:45:39.957408 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:39.957378    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzs4g_cefaeeee-fef2-4a38-a4a1-d26e2b452228/whereabouts-cni/0.log"
Sep 17 23:45:40.216828 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:40.216781    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vvrj6_5ed5a1fa-247e-4efb-b57f-39fd524d7981/kube-multus/0.log"
Sep 17 23:45:40.328733 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:40.328702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-njwnq_f6a51291-55ed-41ea-8029-bf5b677956a8/network-metrics-daemon/0.log"
Sep 17 23:45:40.345235 ip-10-0-139-248 kubenswrapper[2580]: I0917 23:45:40.345207    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-njwnq_f6a51291-55ed-41ea-8029-bf5b677956a8/kube-rbac-proxy/0.log"