Sep 14 10:25:11.326921 ip-10-0-133-125 systemd[1]: Starting Kubernetes Kubelet...
Sep 14 10:25:11.695664 ip-10-0-133-125 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 10:25:11.695664 ip-10-0-133-125 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 14 10:25:11.695664 ip-10-0-133-125 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 10:25:11.695664 ip-10-0-133-125 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 14 10:25:11.695664 ip-10-0-133-125 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 14 10:25:11.697192 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.697105    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 10:25:11.700888 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700873    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:25:11.700888 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700888    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700893    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700896    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700899    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700902    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700904    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700907    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700909    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700911    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700914    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700916    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700919    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700921    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700924    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700926    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700929    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700931    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700934    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700936    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700943    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:25:11.700946 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700945    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700948    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700951    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700954    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700956    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700959    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700961    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700964    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700966    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700970    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700974    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700976    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700979    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700981    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700983    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700986    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700988    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700991    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700993    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:25:11.701397 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700996    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.700998    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701000    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701002    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701005    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701007    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701009    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701012    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701014    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701017    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701019    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701021    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701023    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701026    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701028    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701031    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701034    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701036    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701039    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701042    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:25:11.701902 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701044    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701046    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701048    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701051    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701053    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701056    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701059    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701061    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701063    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701066    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701068    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701071    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701073    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701076    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701079    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701081    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701083    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701085    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701088    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701090    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:25:11.702362 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701095    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701097    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701099    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701102    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701104    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701107    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701111    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701116    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701120    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701122    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701125    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701128    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701130    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701132    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701135    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701137    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701140    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701142    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701144    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:25:11.702834 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701836    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701844    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701847    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701849    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701852    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701855    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701858    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701862    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701865    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701868    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701871    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701874    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701876    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701879    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701882    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701884    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701887    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701890    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701892    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701895    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:25:11.703288 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701897    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701900    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701902    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701905    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701908    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701910    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701912    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701915    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701917    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701920    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701922    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701924    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701927    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701929    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701931    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701933    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701935    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701938    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701941    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701944    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:25:11.703783 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701946    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701949    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701952    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701954    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701956    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701959    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701961    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701963    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701965    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701969    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701972    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701974    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701977    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701980    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701984    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701987    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701990    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701993    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701995    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:25:11.704312 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.701998    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702000    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702002    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702005    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702007    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702011    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702015    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702017    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702020    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702023    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702026    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702028    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702032    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702034    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702037    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702040    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702042    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702044    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702047    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:25:11.704792 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702049    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702051    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702054    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702056    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702059    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702062    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702064    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702066    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702068    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702072    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702074    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702076    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702079    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702081    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702083    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702086    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702088    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702091    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702093    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702096    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:25:11.705252 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702098    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702456    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702460    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702464    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702467    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702473    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702475    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702478    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702481    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702483    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702485    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702488    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702490    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702492    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702495    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702497    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702500    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702503    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702505    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702508    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:25:11.705747 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702510    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702512    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702515    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702517    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702520    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702522    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702525    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702527    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702529    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702532    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702534    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702537    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702539    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702542    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702544    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702547    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702549    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702552    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702555    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702558    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:25:11.706216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702561    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702563    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702580    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702583    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702585    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702588    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702591    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702593    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702596    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702598    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702600    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702603    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702605    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702608    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702610    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702613    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702616    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702620    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702623    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:25:11.706698 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702626    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702629    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702632    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702635    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702638    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702640    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702642    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702645    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702647    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702650    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702653    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702655    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702659    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702661    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702664    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702667    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702669    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702672    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702674    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702676    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:25:11.707135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702679    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702681    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702684    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702686    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702688    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702691    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702693    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702696    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702699    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702702    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702705    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702708    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702710    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702715    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702718    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702720    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702727    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702729    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702731    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:25:11.707613 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702734    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.702736    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702809    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702842    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702850    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702854    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702859    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702862    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702866    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702870    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702873    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702875    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702879    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702882    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702885    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702888    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702890    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702893    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702896    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702899    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702903    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702906    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702909    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702912    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 14 10:25:11.708059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702915    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702918    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702921    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702924    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702927    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702932    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702935    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702938    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702941    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702944    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702948    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702950    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702953    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702956    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702959    2563 flags.go:64] FLAG: --enable-server="true"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702962    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702967    2563 flags.go:64] FLAG: --event-burst="100"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702969    2563 flags.go:64] FLAG: --event-qps="50"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702972    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702975    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702978    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702981    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702984    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702987    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702990    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 14 10:25:11.708679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702993    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702995    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.702998    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703001    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703003    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703006    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703009    2563 flags.go:64] FLAG: --feature-gates=""
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703012    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703015    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703018    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703021    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703023    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703031    2563 flags.go:64] FLAG: --help="false"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703034    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-125.ec2.internal"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703037    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703040    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703042    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703045    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703049    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703051    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703054    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703057    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703059    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 14 10:25:11.709239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703063    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703066    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703068    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703071    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703075    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703077    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703080    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703083    2563 flags.go:64] FLAG: --lock-file=""
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703085    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703088    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703091    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703096    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703098    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703101    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703104    2563 flags.go:64] FLAG: --logging-format="text"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703107    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703110    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703113    2563 flags.go:64] FLAG: --manifest-url=""
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703116    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703119    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703122    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703129    2563 flags.go:64] FLAG: --max-pods="110"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703132    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703135    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703138    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 14 10:25:11.709861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703140    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703143    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703146    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703149    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703155    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703158    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703161    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703164    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703167    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703173    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703175    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703179    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703181    2563 flags.go:64] FLAG: --port="10250"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703184    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703187    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c087a6b8582738e8"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703190    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703193    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703195    2563 flags.go:64] FLAG: --register-node="true"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703198    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703201    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703204    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703207    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703209    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703213    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703216    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 14 10:25:11.710436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703219    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703222    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703224    2563 flags.go:64] FLAG: --runonce="false"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703228    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703232    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703235    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703237    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703240    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703243    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703246    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703249    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703251    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703254    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703257    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703259    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703262    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703265    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703268    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703273    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703276    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703279    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703282    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703285    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703287    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703290    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 14 10:25:11.711043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703293    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703296    2563 flags.go:64] FLAG: --v="2"
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703300    2563 flags.go:64] FLAG: --version="false"
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703303    2563 flags.go:64] FLAG: --vmodule=""
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703307    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.703310    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703416    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703419    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703421    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703424    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703427    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703431    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703434    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703436    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703439    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703441    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703444    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703448    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703451    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703454    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703456    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:25:11.711657 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703459    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703461    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703463    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703467    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703469    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703472    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703474    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703477    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703479    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703481    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703484    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703486    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703489    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703491    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703494    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703496    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703498    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703501    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703503    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:25:11.712254 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703505    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703508    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703511    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703515    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703517    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703520    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703522    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703524    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703527    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703529    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703532    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703534    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703536    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703539    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703541    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703544    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703546    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703555    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703558    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703561    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:25:11.712727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703564    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703582    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703585    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703587    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703590    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703592    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703595    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703599    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703604    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703609    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703614    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703618    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703621    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703625    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703629    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703635    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703639    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703642    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703646    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:25:11.713243 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703650    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703653    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703656    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703660    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703663    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703667    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703671    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703674    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703678    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703681    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703684    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703695    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703701    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703704    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703708    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703711    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703714    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703718    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703721    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703725    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:25:11.713805 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703728    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703731    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703734    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703738    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703741    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.703744    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.704447    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.711928    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.711942    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.711962    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.711967    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712027    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712033    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712036    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:25:11.714272 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712039    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712042    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712044    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712047    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712050    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712052    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712055    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712057    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712060    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712064    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712067    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712069    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712072    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712075    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712078    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712080    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712082    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712085    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712087    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712090    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:25:11.714653 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712093    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712095    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712098    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712100    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712102    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712105    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712109    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712114    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712117    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712120    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712123    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712125    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712128    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712131    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712133    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712136    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712138    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712140    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712143    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:25:11.715325 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712145    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712148    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712150    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712156    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712158    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712161    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712163    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712166    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712168    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712171    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712173    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712176    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712178    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712180    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712183    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712185    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712187    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712190    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712192    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712195    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:25:11.715791 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712198    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712200    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712203    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712205    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712208    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712210    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712213    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712216    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712218    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712221    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712223    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712225    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712228    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712230    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712232    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712235    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712237    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712240    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712242    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712244    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:25:11.716372 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712247    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712249    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712253    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712256    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712259    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712262    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712266    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712268    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712271    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712274    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712276    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712280    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712283    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712286    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712288    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712291    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:25:11.716862 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712293    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.712298    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712410    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712415    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712418    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712421    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712423    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712426    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712429    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712431    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712434    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712436    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712439    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712442    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 14 10:25:11.717277 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712446    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712449    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712452    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712455    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712457    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712459    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712462    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712464    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712467    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712469    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712472    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712474    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712478    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712481    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712483    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712486    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712488    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712490    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712493    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712495    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 14 10:25:11.717691 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712497    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712500    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712502    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712505    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712507    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712509    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712511    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712514    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712516    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712518    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712521    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712523    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712526    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712528    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712530    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712533    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712535    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712538    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712540    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712543    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 14 10:25:11.718164 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712545    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712547    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712550    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712552    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712555    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712557    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712559    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712562    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712564    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712583    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712588    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712592    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712594    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712597    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712599    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712602    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712604    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712606    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712609    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712611    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 14 10:25:11.718644 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712615    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712618    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712621    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712624    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712626    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712629    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712632    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712635    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712638    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712642    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712644    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712647    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712649    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712652    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712654    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712656    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712659    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712662    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712664    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 14 10:25:11.719101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712666    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712669    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712671    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712674    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712676    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712678    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712681    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:11.712683    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.712688    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.712789    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.717000    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.718090    2563 server.go:1019] "Starting client certificate rotation"
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.718200    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 10:25:11.719532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.718233    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 14 10:25:11.720763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.720750    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 10:25:11.722264 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.722246    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 14 10:25:11.731657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.731637    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 14 10:25:11.736690 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.736673    2563 log.go:25] "Validated CRI v1 image API"
Sep 14 10:25:11.737525 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.737510    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 14 10:25:11.739918 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.739901    2563 fs.go:135] Filesystem UUIDs: map[1bc85d0c-9cab-4aa9-9de3-35569ec4788f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 de43a909-c6e0-40d4-b6c0-f09d46e2e869:/dev/nvme0n1p3]
Sep 14 10:25:11.739977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.739918    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 14 10:25:11.744702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.744589    2563 manager.go:216] Machine: {Timestamp:2026-09-14 10:25:11.743397491 +0000 UTC m=+0.298585408 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3204745 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b88fdeed0f2951c5a770cd961493 SystemUUID:ec24b88f-deed-0f29-51c5-a770cd961493 BootID:412b28be-250c-4258-83c7-dfaf8490f0d5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:60:b7:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:60:b7:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:74:40:f0:a8:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 14 10:25:11.744702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.744698    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 14 10:25:11.744826 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.744776    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 14 10:25:11.747096 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.747080    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 10:25:11.747602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.747559    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 10:25:11.747762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.747607    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 14 10:25:11.748920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.748907    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 14 10:25:11.748971 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.748928    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 14 10:25:11.748971 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.748939    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 10:25:11.749614 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.749603    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 14 10:25:11.751040 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.751028    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 14 10:25:11.755755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.755741    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 10:25:11.755885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.755874    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 14 10:25:11.755939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.755912    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 14 10:25:11.755939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.755926    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 10:25:11.756016 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.755946    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 14 10:25:11.756016 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.755960    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 14 10:25:11.756016 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.755972    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 10:25:11.756341 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.756328    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 10:25:11.756400 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.756349    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 14 10:25:11.760145 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.760083    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 14 10:25:11.762419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.762402    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 14 10:25:11.762464 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.762448    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 14 10:25:11.763927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763916    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 14 10:25:11.763973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763933    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 14 10:25:11.763973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763941    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 14 10:25:11.763973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763949    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 14 10:25:11.763973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763961    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 14 10:25:11.763973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763967    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 14 10:25:11.764104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763979    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 14 10:25:11.764104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763985    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 14 10:25:11.764104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763991    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 14 10:25:11.764104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.763996    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 14 10:25:11.764104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.764005    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 14 10:25:11.764104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.764010    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 14 10:25:11.764702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.764693    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 14 10:25:11.764702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.764702    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 14 10:25:11.767147 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.767132    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 14 10:25:11.767199 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.767149    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 14 10:25:11.767421 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.767402    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 14 10:25:11.768561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.768551    2563 server.go:1268] "Started kubelet"
Sep 14 10:25:11.768676 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.768640    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 10:25:11.768753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.768711    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 14 10:25:11.768807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.768779    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 14 10:25:11.769996 ip-10-0-133-125 systemd[1]: Started Kubernetes Kubelet.
Sep 14 10:25:11.770734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.770613    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 14 10:25:11.771247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.771226    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 14 10:25:11.773077 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.773061    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 14 10:25:11.774582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.774550    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cn4b9"
Sep 14 10:25:11.774999 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.774203    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-125.ec2.internal.18d52901209d3f84  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-125.ec2.internal,UID:ip-10-0-133-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-125.ec2.internal,},FirstTimestamp:2026-09-14 10:25:11.768522628 +0000 UTC m=+0.323710545,LastTimestamp:2026-09-14 10:25:11.768522628 +0000 UTC m=+0.323710545,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-125.ec2.internal,}"
Sep 14 10:25:11.775895 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.775880    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 14 10:25:11.776288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.776261    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 10:25:11.776396 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.776383    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:11.777067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777050    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 14 10:25:11.777067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777052    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 14 10:25:11.777187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777089    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 14 10:25:11.777187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777149    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 14 10:25:11.777187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777156    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 14 10:25:11.777263 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777230    2563 factory.go:55] Registering systemd factory
Sep 14 10:25:11.777263 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777243    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 14 10:25:11.777458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777442    2563 factory.go:153] Registering CRI-O factory
Sep 14 10:25:11.777499 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777461    2563 factory.go:223] Registration of the crio container factory successfully
Sep 14 10:25:11.777534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777509    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 14 10:25:11.777534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777531    2563 factory.go:103] Registering Raw factory
Sep 14 10:25:11.777608 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777547    2563 manager.go:1196] Started watching for new ooms in manager
Sep 14 10:25:11.777909 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.777895    2563 manager.go:349] Starting recovery of all containers
Sep 14 10:25:11.778445 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.778415    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 14 10:25:11.778564 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.778543    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 14 10:25:11.784430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.784363    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cn4b9"
Sep 14 10:25:11.784619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.784588    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 14 10:25:11.788368 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.788342    2563 manager.go:354] Recovery completed
Sep 14 10:25:11.795619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.795605    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:25:11.797341 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.797327    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:25:11.797417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.797353    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:25:11.797417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.797363    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:25:11.797863 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.797847    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 14 10:25:11.797863 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.797860    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 14 10:25:11.797982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.797874    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 10:25:11.797982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.797887    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:25:11.798547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.798529    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:25:11.798646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.798556    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:25:11.798646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.798564    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:25:11.799124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.799112    2563 policy_none.go:49] "None policy: Start"
Sep 14 10:25:11.799168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.799128    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 14 10:25:11.799168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.799137    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 14 10:25:11.799746 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.799652    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-125.ec2.internal.18d529012254fe97  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-125.ec2.internal,UID:ip-10-0-133-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-125.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-125.ec2.internal,},FirstTimestamp:2026-09-14 10:25:11.797341847 +0000 UTC m=+0.352529764,LastTimestamp:2026-09-14 10:25:11.797341847 +0000 UTC m=+0.352529764,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-125.ec2.internal,}"
Sep 14 10:25:11.800177 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.800165    2563 policy_none.go:47] "Start"
Sep 14 10:25:11.829179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.829162    2563 manager.go:337] "Starting Device Plugin manager"
Sep 14 10:25:11.829298 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.829219    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 14 10:25:11.829298 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.829234    2563 server.go:87] "Starting device plugin registration server"
Sep 14 10:25:11.829752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.829734    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 14 10:25:11.829839 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.829753    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 14 10:25:11.830270 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.829901    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 14 10:25:11.830270 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.829994    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 14 10:25:11.830270 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.830003    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 14 10:25:11.830444 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.830412    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 14 10:25:11.830492 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.830458    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:11.911733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.911707    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 14 10:25:11.912743 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.911761    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 14 10:25:11.912743 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.911786    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 14 10:25:11.912743 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.911827    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 14 10:25:11.914438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.914422    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:25:11.930596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.930557    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:25:11.931377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.931363    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:25:11.931437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.931391    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:25:11.931437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.931401    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:25:11.931437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.931421    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-125.ec2.internal"
Sep 14 10:25:11.939130 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:11.939112    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-125.ec2.internal"
Sep 14 10:25:11.939175 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.939134    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-125.ec2.internal\": node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:11.983312 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:11.983230    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.011948 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.011925    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"]
Sep 14 10:25:12.012026 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.012013    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:25:12.013497 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.013479    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:25:12.013632 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.013513    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:25:12.013632 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.013526    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:25:12.014857 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.014842    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:25:12.014975 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.014962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.015016 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.014988    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:25:12.015595 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.015559    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:25:12.015673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.015606    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:25:12.015673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.015619    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:25:12.016089 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.016074    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:25:12.016180 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.016104    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:25:12.016180 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.016117    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:25:12.017629 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.017612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.017706 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.017643    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 14 10:25:12.018310 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.018297    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 14 10:25:12.018383 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.018324    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 14 10:25:12.018383 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.018337    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID"
Sep 14 10:25:12.042146 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.042127    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-125.ec2.internal\" not found" node="ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.046464 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.046448    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-125.ec2.internal\" not found" node="ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.078616 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.078594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da429792433b856cd588a17aa37aaa66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"da429792433b856cd588a17aa37aaa66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.078705 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.078621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da429792433b856cd588a17aa37aaa66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"da429792433b856cd588a17aa37aaa66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.078705 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.078640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf677e6df7b80e07cd34581a27ba68d0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"bf677e6df7b80e07cd34581a27ba68d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.083676 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.083659    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.179488 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.179453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da429792433b856cd588a17aa37aaa66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"da429792433b856cd588a17aa37aaa66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.179666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.179495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da429792433b856cd588a17aa37aaa66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"da429792433b856cd588a17aa37aaa66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.179666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.179519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf677e6df7b80e07cd34581a27ba68d0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"bf677e6df7b80e07cd34581a27ba68d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.179666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.179544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da429792433b856cd588a17aa37aaa66-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"da429792433b856cd588a17aa37aaa66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.179666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.179587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da429792433b856cd588a17aa37aaa66-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"da429792433b856cd588a17aa37aaa66\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.179666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.179603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf677e6df7b80e07cd34581a27ba68d0-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"bf677e6df7b80e07cd34581a27ba68d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.184532 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.184517    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.285417 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.285344    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.345528 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.345502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.350003 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.349946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:12.385652 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.385611    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.486160 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.486129    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.586676 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.586606    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.687151 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.687119    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.718293 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.718270    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 14 10:25:12.718673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.718415    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 14 10:25:12.787200 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.787167    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.792333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.792307    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 10:20:12 +0000 UTC" deadline="2028-06-01 07:07:27.693051504 +0000 UTC"
Sep 14 10:25:12.792407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.792334    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15020h42m14.900721347s"
Sep 14 10:25:12.837118 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:12.837065    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda429792433b856cd588a17aa37aaa66.slice/crio-6a210d996ca3c7766f6176ca3f7e240d9d2d79bd26144293c4d1757064cc0c47 WatchSource:0}: Error finding container 6a210d996ca3c7766f6176ca3f7e240d9d2d79bd26144293c4d1757064cc0c47: Status 404 returned error can't find the container with id 6a210d996ca3c7766f6176ca3f7e240d9d2d79bd26144293c4d1757064cc0c47
Sep 14 10:25:12.837319 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:12.837297    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf677e6df7b80e07cd34581a27ba68d0.slice/crio-5f376a451d1a6123b8e3599c2a04eb2f79961c378dc7e0156980f9841e39851d WatchSource:0}: Error finding container 5f376a451d1a6123b8e3599c2a04eb2f79961c378dc7e0156980f9841e39851d: Status 404 returned error can't find the container with id 5f376a451d1a6123b8e3599c2a04eb2f79961c378dc7e0156980f9841e39851d
Sep 14 10:25:12.840729 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.840715    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:25:12.887501 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.887459    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:12.914939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.914878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" event={"ID":"bf677e6df7b80e07cd34581a27ba68d0","Type":"ContainerStarted","Data":"5f376a451d1a6123b8e3599c2a04eb2f79961c378dc7e0156980f9841e39851d"}
Sep 14 10:25:12.915970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.915955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"da429792433b856cd588a17aa37aaa66","Type":"ContainerStarted","Data":"6a210d996ca3c7766f6176ca3f7e240d9d2d79bd26144293c4d1757064cc0c47"}
Sep 14 10:25:12.941021 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.941004    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:25:12.956754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:12.956732    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:25:12.987584 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:12.987548    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:13.088132 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.088064    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found"
Sep 14 10:25:13.174180 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.174151    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:25:13.177804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.177788    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:13.190636 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.190617    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 10:25:13.191672 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.191648    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"
Sep 14 10:25:13.201031 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.201014    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 14 10:25:13.673775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.673740    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 14 10:25:13.757341 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.757309    2563 apiserver.go:52] "Watching apiserver"
Sep 14 10:25:13.765562 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.765528    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 14 10:25:13.768079 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.768055    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-hwrdf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal","openshift-multus/multus-6xzkn","openshift-multus/multus-additional-cni-plugins-f4hh7","openshift-multus/network-metrics-daemon-pq8l7","openshift-network-diagnostics/network-check-target-mx88d","kube-system/global-pull-secret-syncer-tbshl","kube-system/konnectivity-agent-rvl4z","kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b","openshift-dns/node-resolver-5l8m9","openshift-network-operator/iptables-alerter-4xk5c","openshift-ovn-kubernetes/ovnkube-node-j556p","openshift-cluster-node-tuning-operator/tuned-vwdsr"]
Sep 14 10:25:13.769713 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.769689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:13.770968 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.770941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.772201 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.772178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.772734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.772714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2kppw\""
Sep 14 10:25:13.772734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.772717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 14 10:25:13.772902 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.772807    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qv9v6\""
Sep 14 10:25:13.772902 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.772818    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 14 10:25:13.773318 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.773300    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:25:13.773585 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.773540    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 14 10:25:13.773809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.773791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.773923 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.773873    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:13.774507 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.774489    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:25:13.774614 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.774602    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 14 10:25:13.775208 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.775018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:13.775208 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.775094    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:13.775208 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.775199    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 14 10:25:13.775398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.775280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b7nj6\""
Sep 14 10:25:13.776520 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.776334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:13.776520 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.776413    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:13.776983 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.776963    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 14 10:25:13.778791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.778402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.780153 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.780137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.780786 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.780766    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 14 10:25:13.780923 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.780803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 14 10:25:13.781130 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.781111    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 14 10:25:13.781204 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.781148    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4nms4\""
Sep 14 10:25:13.782106 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.782089    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 14 10:25:13.782106 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.782099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.782772 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.782753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 14 10:25:13.783238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.783058    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 14 10:25:13.783238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.783098    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 14 10:25:13.783238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.783111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrxx4\""
Sep 14 10:25:13.783515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.783492    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 14 10:25:13.783899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.783877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.784275 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.784255    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 14 10:25:13.784513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.784496    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 14 10:25:13.784710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.784502    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 14 10:25:13.784805 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.784787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2l6tm\""
Sep 14 10:25:13.785514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.785496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.786013 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.785995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 14 10:25:13.786345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.786328    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 14 10:25:13.786649 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.786631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v9pk9\""
Sep 14 10:25:13.787178 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.787156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.788128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-lib-modules\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f1aa06-09e0-4f42-ad79-529290b9db2c-serviceca\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.788220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmsw\" (UniqueName: \"kubernetes.io/projected/b0f1aa06-09e0-4f42-ad79-529290b9db2c-kube-api-access-qhmsw\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.788220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-systemd\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59facbf-7d67-430f-a4c9-38456ddf76d3-iptables-alerter-script\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.788424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-tuned\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b6fabe4-b972-42d0-b1c9-306df81cbb62-konnectivity-ca\") pod \"konnectivity-agent-rvl4z\" (UID: \"9b6fabe4-b972-42d0-b1c9-306df81cbb62\") " pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:13.788424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-modprobe-d\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-sys\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59facbf-7d67-430f-a4c9-38456ddf76d3-host-slash\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f1aa06-09e0-4f42-ad79-529290b9db2c-host\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysconfig\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-run\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m7sw\" (UniqueName: \"kubernetes.io/projected/b41dccc1-059f-48f1-8a80-453ce7108d65-kube-api-access-7m7sw\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t66q\" (UniqueName: \"kubernetes.io/projected/a59facbf-7d67-430f-a4c9-38456ddf76d3-kube-api-access-2t66q\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b35dd04-3950-42a5-9755-03cb4f5059b6-dbus\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-var-lib-kubelet\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-host\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-kubernetes\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.788701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b35dd04-3950-42a5-9755-03cb4f5059b6-kubelet-config\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzkf\" (UniqueName: \"kubernetes.io/projected/c857632f-81d2-46e9-b1fb-f6921245a682-kube-api-access-grzkf\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysctl-d\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysctl-conf\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b41dccc1-059f-48f1-8a80-453ce7108d65-tmp\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b6fabe4-b972-42d0-b1c9-306df81cbb62-agent-certs\") pod \"konnectivity-agent-rvl4z\" (UID: \"9b6fabe4-b972-42d0-b1c9-306df81cbb62\") " pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:13.789104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.788910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:13.789592 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.789557    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 14 10:25:13.789703 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.789665    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9zl2s\""
Sep 14 10:25:13.794582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.794403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 14 10:25:13.794582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.794415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 14 10:25:13.794582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.794445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 14 10:25:13.794582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.794486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r29bk\""
Sep 14 10:25:13.794582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.794517    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 14 10:25:13.794582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.794564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 14 10:25:13.794899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.794822    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 14 10:25:13.803067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.803042    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 14 10:25:13.873026 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.872941    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5tgwj"
Sep 14 10:25:13.879011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.878981    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 14 10:25:13.880072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.880051    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5tgwj"
Sep 14 10:25:13.889385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-sys\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.889487 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f1aa06-09e0-4f42-ad79-529290b9db2c-host\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.889487 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-systemd-units\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.889487 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.889487 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-cni-netd\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-sys\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-sys-fs\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f1aa06-09e0-4f42-ad79-529290b9db2c-host\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysconfig\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m7sw\" (UniqueName: \"kubernetes.io/projected/b41dccc1-059f-48f1-8a80-453ce7108d65-kube-api-access-7m7sw\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-kubelet\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysconfig\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-ovnkube-config\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-kubelet\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.889707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-ovnkube-script-lib\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l956l\" (UniqueName: \"kubernetes.io/projected/462c8353-8c49-416b-9cf4-3a5c680574d6-kube-api-access-l956l\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b35dd04-3950-42a5-9755-03cb4f5059b6-kubelet-config\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-system-cni-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b35dd04-3950-42a5-9755-03cb4f5059b6-kubelet-config\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-var-lib-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxjc7\" (UniqueName: \"kubernetes.io/projected/a18d0c66-674c-45fc-8fbe-d283baeeb203-kube-api-access-wxjc7\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b6fabe4-b972-42d0-b1c9-306df81cbb62-agent-certs\") pod \"konnectivity-agent-rvl4z\" (UID: \"9b6fabe4-b972-42d0-b1c9-306df81cbb62\") " pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8dz\" (UniqueName: \"kubernetes.io/projected/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-kube-api-access-lg8dz\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.889986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-hosts-file\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxk5l\" (UniqueName: \"kubernetes.io/projected/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-kube-api-access-cxk5l\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-daemon-config\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-systemd\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-log-socket\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.890151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-systemd\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-cni-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-systemd\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-system-cni-dir\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-cnibin\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-k8s-cni-cncf-io\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-modprobe-d\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59facbf-7d67-430f-a4c9-38456ddf76d3-host-slash\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-cni-binary-copy\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-socket-dir-parent\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-modprobe-d\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a59facbf-7d67-430f-a4c9-38456ddf76d3-host-slash\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-netns\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-cni-bin\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.890901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890645    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-etc-kubernetes\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-run\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t66q\" (UniqueName: \"kubernetes.io/projected/a59facbf-7d67-430f-a4c9-38456ddf76d3-kube-api-access-2t66q\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b35dd04-3950-42a5-9755-03cb4f5059b6-dbus\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-run\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-conf-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-multus-certs\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b35dd04-3950-42a5-9755-03cb4f5059b6-dbus\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-etc-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-node-log\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-cni-binary-copy\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-var-lib-kubelet\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-host\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-kubernetes\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.890999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-var-lib-kubelet\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.891702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-device-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-host\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzkf\" (UniqueName: \"kubernetes.io/projected/c857632f-81d2-46e9-b1fb-f6921245a682-kube-api-access-grzkf\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/462c8353-8c49-416b-9cf4-3a5c680574d6-ovn-node-metrics-cert\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-kubernetes\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.891246    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.891344    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret podName:7b35dd04-3950-42a5-9755-03cb4f5059b6 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:14.391320407 +0000 UTC m=+2.946508334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret") pod "global-pull-secret-syncer-tbshl" (UID: "7b35dd04-3950-42a5-9755-03cb4f5059b6") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-socket-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-etc-selinux\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysctl-d\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysctl-conf\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b41dccc1-059f-48f1-8a80-453ce7108d65-tmp\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:13.892430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-cnibin\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8544h\" (UniqueName: \"kubernetes.io/projected/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-kube-api-access-8544h\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-slash\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysctl-d\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-sysctl-conf\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-env-overrides\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-lib-modules\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.891728    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f1aa06-09e0-4f42-ad79-529290b9db2c-serviceca\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmsw\" (UniqueName: \"kubernetes.io/projected/b0f1aa06-09e0-4f42-ad79-529290b9db2c-kube-api-access-qhmsw\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-ovn\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b41dccc1-059f-48f1-8a80-453ce7108d65-lib-modules\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-cni-bin\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.891949    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs podName:c857632f-81d2-46e9-b1fb-f6921245a682 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:14.391932415 +0000 UTC m=+2.947120332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs") pod "network-metrics-daemon-pq8l7" (UID: "c857632f-81d2-46e9-b1fb-f6921245a682") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.891971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-tmp-dir\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59facbf-7d67-430f-a4c9-38456ddf76d3-iptables-alerter-script\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-cni-multus\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.893247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-hostroot\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f1aa06-09e0-4f42-ad79-529290b9db2c-serviceca\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-tuned\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b6fabe4-b972-42d0-b1c9-306df81cbb62-konnectivity-ca\") pod \"konnectivity-agent-rvl4z\" (UID: \"9b6fabe4-b972-42d0-b1c9-306df81cbb62\") " pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-run-netns\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-registration-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.892659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a59facbf-7d67-430f-a4c9-38456ddf76d3-iptables-alerter-script\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.894117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.893074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b6fabe4-b972-42d0-b1c9-306df81cbb62-konnectivity-ca\") pod \"konnectivity-agent-rvl4z\" (UID: \"9b6fabe4-b972-42d0-b1c9-306df81cbb62\") " pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:13.894976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.894956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b41dccc1-059f-48f1-8a80-453ce7108d65-tmp\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.895072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.895043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b41dccc1-059f-48f1-8a80-453ce7108d65-etc-tuned\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.895395 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.895377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b6fabe4-b972-42d0-b1c9-306df81cbb62-agent-certs\") pod \"konnectivity-agent-rvl4z\" (UID: \"9b6fabe4-b972-42d0-b1c9-306df81cbb62\") " pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:13.904394 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.904373    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:25:13.904497 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.904398    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:25:13.904497 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.904412    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5h2nx for pod openshift-network-diagnostics/network-check-target-mx88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:13.904497 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:13.904479    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx podName:7144581f-3302-4520-aade-92b01e5f7058 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:14.404462303 +0000 UTC m=+2.959650216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5h2nx" (UniqueName: "kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx") pod "network-check-target-mx88d" (UID: "7144581f-3302-4520-aade-92b01e5f7058") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:13.905015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.904992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t66q\" (UniqueName: \"kubernetes.io/projected/a59facbf-7d67-430f-a4c9-38456ddf76d3-kube-api-access-2t66q\") pod \"iptables-alerter-4xk5c\" (UID: \"a59facbf-7d67-430f-a4c9-38456ddf76d3\") " pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:13.905133 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.905032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m7sw\" (UniqueName: \"kubernetes.io/projected/b41dccc1-059f-48f1-8a80-453ce7108d65-kube-api-access-7m7sw\") pod \"tuned-vwdsr\" (UID: \"b41dccc1-059f-48f1-8a80-453ce7108d65\") " pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:13.907657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.907611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmsw\" (UniqueName: \"kubernetes.io/projected/b0f1aa06-09e0-4f42-ad79-529290b9db2c-kube-api-access-qhmsw\") pod \"node-ca-hwrdf\" (UID: \"b0f1aa06-09e0-4f42-ad79-529290b9db2c\") " pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:13.907876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.907859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzkf\" (UniqueName: \"kubernetes.io/projected/c857632f-81d2-46e9-b1fb-f6921245a682-kube-api-access-grzkf\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-ovn\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-cni-bin\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-tmp-dir\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-cni-multus\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-ovn\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-hostroot\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-hostroot\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-run-netns\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-cni-bin\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-registration-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-systemd-units\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-cni-netd\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-sys-fs\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-cni-multus\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-kubelet\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.993657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-ovnkube-config\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.994601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-kubelet\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.994601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-ovnkube-script-lib\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.994601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-run-netns\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.994601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l956l\" (UniqueName: \"kubernetes.io/projected/462c8353-8c49-416b-9cf4-3a5c680574d6-kube-api-access-l956l\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.994601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-system-cni-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.994601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-var-lib-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.994888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-tmp-dir\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.994983 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxjc7\" (UniqueName: \"kubernetes.io/projected/a18d0c66-674c-45fc-8fbe-d283baeeb203-kube-api-access-wxjc7\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.994983 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.994953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8dz\" (UniqueName: \"kubernetes.io/projected/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-kube-api-access-lg8dz\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.994983 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-registration-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.994983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-hosts-file\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-var-lib-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-cni-netd\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-sys-fs\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-system-cni-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxk5l\" (UniqueName: \"kubernetes.io/projected/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-kube-api-access-cxk5l\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-daemon-config\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.994586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-ovnkube-script-lib\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.994633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-ovnkube-config\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-systemd-units\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-kubelet\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-kubelet\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-hosts-file\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:13.995124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.993483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-systemd\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-log-socket\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-log-socket\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-cni-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-cni-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-run-systemd\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-system-cni-dir\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-cnibin\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-k8s-cni-cncf-io\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-system-cni-dir\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-cnibin\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-daemon-config\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.995800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-cni-binary-copy\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-k8s-cni-cncf-io\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-socket-dir-parent\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-netns\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-cni-bin\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-socket-dir-parent\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-netns\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-etc-kubernetes\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-var-lib-cni-bin\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-conf-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-multus-certs\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-etc-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-node-log\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-multus-conf-dir\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-cni-binary-copy\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-etc-kubernetes\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-host-run-multus-certs\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.996580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-device-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-etc-openvswitch\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995816    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-node-log\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/462c8353-8c49-416b-9cf4-3a5c680574d6-ovn-node-metrics-cert\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-socket-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-etc-selinux\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a18d0c66-674c-45fc-8fbe-d283baeeb203-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-cnibin\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8544h\" (UniqueName: \"kubernetes.io/projected/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-kube-api-access-8544h\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-slash\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-env-overrides\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-socket-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.995990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-device-dir\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-cnibin\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.997428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/462c8353-8c49-416b-9cf4-3a5c680574d6-host-slash\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.998099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-etc-selinux\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:13.998099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-cni-binary-copy\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.998099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/462c8353-8c49-416b-9cf4-3a5c680574d6-env-overrides\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:13.998099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a18d0c66-674c-45fc-8fbe-d283baeeb203-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:13.998099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.996696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-cni-binary-copy\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:13.998623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:13.998595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/462c8353-8c49-416b-9cf4-3a5c680574d6-ovn-node-metrics-cert\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:14.003724 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.003699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l956l\" (UniqueName: \"kubernetes.io/projected/462c8353-8c49-416b-9cf4-3a5c680574d6-kube-api-access-l956l\") pod \"ovnkube-node-j556p\" (UID: \"462c8353-8c49-416b-9cf4-3a5c680574d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:14.004060 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.004032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxjc7\" (UniqueName: \"kubernetes.io/projected/a18d0c66-674c-45fc-8fbe-d283baeeb203-kube-api-access-wxjc7\") pod \"multus-additional-cni-plugins-f4hh7\" (UID: \"a18d0c66-674c-45fc-8fbe-d283baeeb203\") " pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:14.007069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.007043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8dz\" (UniqueName: \"kubernetes.io/projected/9c2fdbe1-ea25-4621-9ac4-da1ac70f60be-kube-api-access-lg8dz\") pod \"aws-ebs-csi-driver-node-wm96b\" (UID: \"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:14.007563 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.007538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxk5l\" (UniqueName: \"kubernetes.io/projected/e0aaadf1-3ea1-463d-b305-7fa7999dcfaf-kube-api-access-cxk5l\") pod \"node-resolver-5l8m9\" (UID: \"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf\") " pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:14.007828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.007785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8544h\" (UniqueName: \"kubernetes.io/projected/b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2-kube-api-access-8544h\") pod \"multus-6xzkn\" (UID: \"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2\") " pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:14.082510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.082301    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:14.090633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.090599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vwdsr"
Sep 14 10:25:14.098680 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.098660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xk5c"
Sep 14 10:25:14.104547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.104528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hwrdf"
Sep 14 10:25:14.111322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.111303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f4hh7"
Sep 14 10:25:14.117998 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.117976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b"
Sep 14 10:25:14.124783 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.124762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5l8m9"
Sep 14 10:25:14.131574 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.131556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6xzkn"
Sep 14 10:25:14.138394 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.138375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:14.399615 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.399545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:14.399615 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.399582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:14.399821 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.399721    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:14.399821 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.399722    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:14.399821 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.399779    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs podName:c857632f-81d2-46e9-b1fb-f6921245a682 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:15.399763831 +0000 UTC m=+3.954951738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs") pod "network-metrics-daemon-pq8l7" (UID: "c857632f-81d2-46e9-b1fb-f6921245a682") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:14.399821 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.399795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret podName:7b35dd04-3950-42a5-9755-03cb4f5059b6 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:15.399788536 +0000 UTC m=+3.954976442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret") pod "global-pull-secret-syncer-tbshl" (UID: "7b35dd04-3950-42a5-9755-03cb4f5059b6") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:14.500519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.500481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:14.500732 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.500700    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:25:14.500732 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.500723    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:25:14.500732 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.500733    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5h2nx for pod openshift-network-diagnostics/network-check-target-mx88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:14.500887 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.500808    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx podName:7144581f-3302-4520-aade-92b01e5f7058 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:15.50078901 +0000 UTC m=+4.055976941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5h2nx" (UniqueName: "kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx") pod "network-check-target-mx88d" (UID: "7144581f-3302-4520-aade-92b01e5f7058") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:14.760176 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.760151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2fdbe1_ea25_4621_9ac4_da1ac70f60be.slice/crio-7ee0a209f597510f43af1d3c285859fa38046bd966be08dba28423f743ac1f8a WatchSource:0}: Error finding container 7ee0a209f597510f43af1d3c285859fa38046bd966be08dba28423f743ac1f8a: Status 404 returned error can't find the container with id 7ee0a209f597510f43af1d3c285859fa38046bd966be08dba28423f743ac1f8a
Sep 14 10:25:14.761971 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.761906    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb820a1e5_95bd_4e5e_a5eb_b42b045ef0f2.slice/crio-c012fcb01ff5195a66ac1eb4f7776334b7c62e1fb399d17b70b8a4dd07db841c WatchSource:0}: Error finding container c012fcb01ff5195a66ac1eb4f7776334b7c62e1fb399d17b70b8a4dd07db841c: Status 404 returned error can't find the container with id c012fcb01ff5195a66ac1eb4f7776334b7c62e1fb399d17b70b8a4dd07db841c
Sep 14 10:25:14.762873 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.762851    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f1aa06_09e0_4f42_ad79_529290b9db2c.slice/crio-59dbec0af15712dd64bd97eefccb5c16dfa8a2a884b12b1ffebabfb99035988b WatchSource:0}: Error finding container 59dbec0af15712dd64bd97eefccb5c16dfa8a2a884b12b1ffebabfb99035988b: Status 404 returned error can't find the container with id 59dbec0af15712dd64bd97eefccb5c16dfa8a2a884b12b1ffebabfb99035988b
Sep 14 10:25:14.763626 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.763603    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6fabe4_b972_42d0_b1c9_306df81cbb62.slice/crio-9f0104c22b4f107f62d81e36a6558f93c628a9d9321f50831db2f89f0f4024fc WatchSource:0}: Error finding container 9f0104c22b4f107f62d81e36a6558f93c628a9d9321f50831db2f89f0f4024fc: Status 404 returned error can't find the container with id 9f0104c22b4f107f62d81e36a6558f93c628a9d9321f50831db2f89f0f4024fc
Sep 14 10:25:14.764460 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.764356    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41dccc1_059f_48f1_8a80_453ce7108d65.slice/crio-82f97d50dbdb9804866a3f6ab366da840d654f13d635aeb98ee8e2417ec2a01a WatchSource:0}: Error finding container 82f97d50dbdb9804866a3f6ab366da840d654f13d635aeb98ee8e2417ec2a01a: Status 404 returned error can't find the container with id 82f97d50dbdb9804866a3f6ab366da840d654f13d635aeb98ee8e2417ec2a01a
Sep 14 10:25:14.767195 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.767172    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462c8353_8c49_416b_9cf4_3a5c680574d6.slice/crio-cd98ddbc14ee67c198c938bfe9d0ef445c1a3d60a28f9368305e09911f734877 WatchSource:0}: Error finding container cd98ddbc14ee67c198c938bfe9d0ef445c1a3d60a28f9368305e09911f734877: Status 404 returned error can't find the container with id cd98ddbc14ee67c198c938bfe9d0ef445c1a3d60a28f9368305e09911f734877
Sep 14 10:25:14.769341 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.769312    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59facbf_7d67_430f_a4c9_38456ddf76d3.slice/crio-c4b3cb2b73859fff1513a104b5123d560d71bc9693979ca8588d19c59ae21347 WatchSource:0}: Error finding container c4b3cb2b73859fff1513a104b5123d560d71bc9693979ca8588d19c59ae21347: Status 404 returned error can't find the container with id c4b3cb2b73859fff1513a104b5123d560d71bc9693979ca8588d19c59ae21347
Sep 14 10:25:14.770398 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.770371    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aaadf1_3ea1_463d_b305_7fa7999dcfaf.slice/crio-a9e2e4c829a245562e969a8cf9f047ea54e06291a86c42fd36dee016179dbe42 WatchSource:0}: Error finding container a9e2e4c829a245562e969a8cf9f047ea54e06291a86c42fd36dee016179dbe42: Status 404 returned error can't find the container with id a9e2e4c829a245562e969a8cf9f047ea54e06291a86c42fd36dee016179dbe42
Sep 14 10:25:14.770983 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:14.770958    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18d0c66_674c_45fc_8fbe_d283baeeb203.slice/crio-f68d2c86ef9cab14a82d1514527442e7a318e0611089ad1a7214d34eaf2ccd70 WatchSource:0}: Error finding container f68d2c86ef9cab14a82d1514527442e7a318e0611089ad1a7214d34eaf2ccd70: Status 404 returned error can't find the container with id f68d2c86ef9cab14a82d1514527442e7a318e0611089ad1a7214d34eaf2ccd70
Sep 14 10:25:14.882939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.882895    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:20:14 +0000 UTC" deadline="2028-03-29 11:54:25.760407512 +0000 UTC"
Sep 14 10:25:14.882939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.882934    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13489h29m10.877477862s"
Sep 14 10:25:14.912834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.912807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:14.912960 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.912930    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:14.913063 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.912987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:14.913139 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:14.913116    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:14.923334 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.923305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vwdsr" event={"ID":"b41dccc1-059f-48f1-8a80-453ce7108d65","Type":"ContainerStarted","Data":"82f97d50dbdb9804866a3f6ab366da840d654f13d635aeb98ee8e2417ec2a01a"}
Sep 14 10:25:14.924746 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.924717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rvl4z" event={"ID":"9b6fabe4-b972-42d0-b1c9-306df81cbb62","Type":"ContainerStarted","Data":"9f0104c22b4f107f62d81e36a6558f93c628a9d9321f50831db2f89f0f4024fc"}
Sep 14 10:25:14.925969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.925948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"cd98ddbc14ee67c198c938bfe9d0ef445c1a3d60a28f9368305e09911f734877"}
Sep 14 10:25:14.927202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.927174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hwrdf" event={"ID":"b0f1aa06-09e0-4f42-ad79-529290b9db2c","Type":"ContainerStarted","Data":"59dbec0af15712dd64bd97eefccb5c16dfa8a2a884b12b1ffebabfb99035988b"}
Sep 14 10:25:14.928467 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.928441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6xzkn" event={"ID":"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2","Type":"ContainerStarted","Data":"c012fcb01ff5195a66ac1eb4f7776334b7c62e1fb399d17b70b8a4dd07db841c"}
Sep 14 10:25:14.929796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.929771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b" event={"ID":"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be","Type":"ContainerStarted","Data":"7ee0a209f597510f43af1d3c285859fa38046bd966be08dba28423f743ac1f8a"}
Sep 14 10:25:14.931874 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.931855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" event={"ID":"bf677e6df7b80e07cd34581a27ba68d0","Type":"ContainerStarted","Data":"d4d4f8d1bee64b45dba4d3969b79357e0f55a79d0745f809fb6fa05436f0059a"}
Sep 14 10:25:14.933266 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.933244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerStarted","Data":"f68d2c86ef9cab14a82d1514527442e7a318e0611089ad1a7214d34eaf2ccd70"}
Sep 14 10:25:14.934403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.934382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5l8m9" event={"ID":"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf","Type":"ContainerStarted","Data":"a9e2e4c829a245562e969a8cf9f047ea54e06291a86c42fd36dee016179dbe42"}
Sep 14 10:25:14.935431 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.935410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xk5c" event={"ID":"a59facbf-7d67-430f-a4c9-38456ddf76d3","Type":"ContainerStarted","Data":"c4b3cb2b73859fff1513a104b5123d560d71bc9693979ca8588d19c59ae21347"}
Sep 14 10:25:14.953681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:14.953607    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" podStartSLOduration=0.95359114 podStartE2EDuration="953.59114ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:14.953421148 +0000 UTC m=+3.508609104" watchObservedRunningTime="2026-09-14 10:25:14.95359114 +0000 UTC m=+3.508779072"
Sep 14 10:25:15.408312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.408273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:15.408508 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.408328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:15.408508 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.408502    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:15.408614 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.408568    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs podName:c857632f-81d2-46e9-b1fb-f6921245a682 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:17.408548693 +0000 UTC m=+5.963736614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs") pod "network-metrics-daemon-pq8l7" (UID: "c857632f-81d2-46e9-b1fb-f6921245a682") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:15.409016 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.408996    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:15.409118 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.409048    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret podName:7b35dd04-3950-42a5-9755-03cb4f5059b6 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:17.409033332 +0000 UTC m=+5.964221253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret") pod "global-pull-secret-syncer-tbshl" (UID: "7b35dd04-3950-42a5-9755-03cb4f5059b6") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:15.509428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.508981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:15.509428 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.509200    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:25:15.509428 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.509219    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:25:15.509428 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.509232    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5h2nx for pod openshift-network-diagnostics/network-check-target-mx88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:15.509428 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.509290    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx podName:7144581f-3302-4520-aade-92b01e5f7058 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:17.509272086 +0000 UTC m=+6.064459995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5h2nx" (UniqueName: "kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx") pod "network-check-target-mx88d" (UID: "7144581f-3302-4520-aade-92b01e5f7058") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:15.883505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.883371    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:20:14 +0000 UTC" deadline="2028-04-19 10:26:29.411665931 +0000 UTC"
Sep 14 10:25:15.883505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.883432    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13992h1m13.528237903s"
Sep 14 10:25:15.913834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.913778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:15.913988 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:15.913918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:15.955036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.954998    2563 generic.go:356] "Generic (PLEG): container finished" podID="da429792433b856cd588a17aa37aaa66" containerID="a86ab780f342d8aa6231192d62b7a05d296348f8419f67f86c873b0c62e67128" exitCode=0
Sep 14 10:25:15.955768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:15.955693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"da429792433b856cd588a17aa37aaa66","Type":"ContainerDied","Data":"a86ab780f342d8aa6231192d62b7a05d296348f8419f67f86c873b0c62e67128"}
Sep 14 10:25:16.913117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:16.912369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:16.913117 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:16.912506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:16.913117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:16.912956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:16.913117 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:16.913063    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:16.975409 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:16.975371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"da429792433b856cd588a17aa37aaa66","Type":"ContainerStarted","Data":"ccfb0fb72adb99f5d42d2c0457b8e91a94f44e07a2968b25acff0bf076644ac9"}
Sep 14 10:25:17.431890 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:17.431121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:17.431890 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:17.431186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:17.431890 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.431296    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:17.431890 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.431357    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret podName:7b35dd04-3950-42a5-9755-03cb4f5059b6 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:21.431339996 +0000 UTC m=+9.986527905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret") pod "global-pull-secret-syncer-tbshl" (UID: "7b35dd04-3950-42a5-9755-03cb4f5059b6") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:17.431890 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.431799    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:17.431890 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.431852    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs podName:c857632f-81d2-46e9-b1fb-f6921245a682 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:21.431835495 +0000 UTC m=+9.987023407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs") pod "network-metrics-daemon-pq8l7" (UID: "c857632f-81d2-46e9-b1fb-f6921245a682") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:17.532079 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:17.532031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:17.532274 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.532233    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:25:17.532274 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.532253    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:25:17.532274 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.532266    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5h2nx for pod openshift-network-diagnostics/network-check-target-mx88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:17.532432 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.532330    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx podName:7144581f-3302-4520-aade-92b01e5f7058 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:21.532311677 +0000 UTC m=+10.087499586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5h2nx" (UniqueName: "kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx") pod "network-check-target-mx88d" (UID: "7144581f-3302-4520-aade-92b01e5f7058") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:17.916642 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:17.916605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:17.917105 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:17.916820    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:18.913043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:18.913007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:18.913226 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:18.913139    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:18.913226 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:18.913210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:18.913362 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:18.913300    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:19.914510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:19.913702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:19.914510 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:19.913863    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:20.912868 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:20.912831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:20.913054 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:20.912978    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:20.913388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:20.913369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:20.913505 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:20.913465    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:21.469918 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:21.469159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:21.469918 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:21.469217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:21.469918 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.469374    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:21.469918 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.469444    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs podName:c857632f-81d2-46e9-b1fb-f6921245a682 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:29.469428161 +0000 UTC m=+18.024616086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs") pod "network-metrics-daemon-pq8l7" (UID: "c857632f-81d2-46e9-b1fb-f6921245a682") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:21.469918 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.469829    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:21.469918 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.469880    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret podName:7b35dd04-3950-42a5-9755-03cb4f5059b6 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:29.469864617 +0000 UTC m=+18.025052531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret") pod "global-pull-secret-syncer-tbshl" (UID: "7b35dd04-3950-42a5-9755-03cb4f5059b6") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:21.569644 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:21.569583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:21.569855 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.569834    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:25:21.569926 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.569867    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:25:21.569926 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.569880    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5h2nx for pod openshift-network-diagnostics/network-check-target-mx88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:21.570029 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.569944    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx podName:7144581f-3302-4520-aade-92b01e5f7058 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:29.56992473 +0000 UTC m=+18.125112654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5h2nx" (UniqueName: "kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx") pod "network-check-target-mx88d" (UID: "7144581f-3302-4520-aade-92b01e5f7058") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:21.914448 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:21.913933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:21.914448 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:21.914076    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:22.912507 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:22.912256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:22.913000 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:22.912587    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:22.913000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:22.912906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:22.913000 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:22.912984    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:23.913605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:23.913027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:23.913605 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:23.913224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:24.912342 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:24.912297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:24.912524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:24.912320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:24.912524 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:24.912445    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:24.912524 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:24.912500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:25.912451 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:25.912420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:25.912805 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:25.912527    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:26.912631 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:26.912601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:26.913008 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:26.912708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:26.913008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:26.912772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:26.913008 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:26.912891    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:27.912256 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:27.912219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:27.912437 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:27.912370    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:28.912911 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:28.912873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:28.913424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:28.912887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:28.913424 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:28.912998    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:28.913424 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:28.913093    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:29.535307 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:29.535267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:29.535307 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:29.535314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:29.535544 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.535405    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:29.535544 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.535433    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:29.535544 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.535474    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret podName:7b35dd04-3950-42a5-9755-03cb4f5059b6 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:45.535456533 +0000 UTC m=+34.090644440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret") pod "global-pull-secret-syncer-tbshl" (UID: "7b35dd04-3950-42a5-9755-03cb4f5059b6") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:29.535544 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.535494    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs podName:c857632f-81d2-46e9-b1fb-f6921245a682 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:45.535482776 +0000 UTC m=+34.090670680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs") pod "network-metrics-daemon-pq8l7" (UID: "c857632f-81d2-46e9-b1fb-f6921245a682") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:29.636166 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:29.636139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:29.636313 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.636294    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:25:29.636313 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.636312    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:25:29.636389 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.636321    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5h2nx for pod openshift-network-diagnostics/network-check-target-mx88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:29.636389 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.636368    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx podName:7144581f-3302-4520-aade-92b01e5f7058 nodeName:}" failed. No retries permitted until 2026-09-14 10:25:45.636354844 +0000 UTC m=+34.191542748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5h2nx" (UniqueName: "kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx") pod "network-check-target-mx88d" (UID: "7144581f-3302-4520-aade-92b01e5f7058") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:29.912072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:29.912038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:29.912235 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:29.912148    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:30.912656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:30.912617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:30.913080 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:30.912629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:30.913080 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:30.912824    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:30.913080 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:30.912958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:31.913328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:31.913294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:31.913799 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:31.913402    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:32.912667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:32.912634    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:32.912667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:32.912655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:32.912865 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:32.912755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:32.912904 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:32.912854    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:33.912011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:33.911983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:33.912968 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:33.912133    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:34.024017 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.023936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rvl4z" event={"ID":"9b6fabe4-b972-42d0-b1c9-306df81cbb62","Type":"ContainerStarted","Data":"743004435fcdec495c821916e2b815e0edfc6910b11a0ab5ceb3aef4465110cd"}
Sep 14 10:25:34.028174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.028140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"203e0ac85be48bba11e6a5467ad463f61bbbfadcc7ce8a6cc05931aff47e09b4"}
Sep 14 10:25:34.028275 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.028232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"98f5e0952a62626b4d784a8dba338bd838a4496b12cdae04342bf2133ae8d194"}
Sep 14 10:25:34.028308 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.028281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"986e09ac9965f5ee414c2c5b577952b78a3b1512aae8003e6609a7ed3add9820"}
Sep 14 10:25:34.028346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.028326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"6f94e0a5fcb88eb0e9f45041843afb402878bd339bb4a58170dedc0a9c581dc1"}
Sep 14 10:25:34.028382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.028371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"bd37d5f2ff6a3395bb3dcac83ea38c8fe7f77dda9639eaf9ecb7fe35361cc457"}
Sep 14 10:25:34.028427 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.028417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"c2e76d20e8a0a621f58f640df3180e8f6f916aaf630732e3a53ee1155a26d0d3"}
Sep 14 10:25:34.029783 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.029753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hwrdf" event={"ID":"b0f1aa06-09e0-4f42-ad79-529290b9db2c","Type":"ContainerStarted","Data":"d0356df69329a998eb063a3211d9f0dd5c28dd3c5d71bfd4c5985b1c51f2dce5"}
Sep 14 10:25:34.031448 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.031405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6xzkn" event={"ID":"b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2","Type":"ContainerStarted","Data":"31413b2acadc16dbf4cb4d967d2b01101a1f3b756160818302add0e0600bacb2"}
Sep 14 10:25:34.033114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.033091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b" event={"ID":"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be","Type":"ContainerStarted","Data":"a1bd3aee84cddf6cfa4fcd809a2af156a069060e964692bdee7fb918931c3ea1"}
Sep 14 10:25:34.034640 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.034616    2563 generic.go:356] "Generic (PLEG): container finished" podID="a18d0c66-674c-45fc-8fbe-d283baeeb203" containerID="8112382caff4f6709aa335d8560b265c71cd2ed8d1ed0f991af2127509163e52" exitCode=0
Sep 14 10:25:34.034727 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.034684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerDied","Data":"8112382caff4f6709aa335d8560b265c71cd2ed8d1ed0f991af2127509163e52"}
Sep 14 10:25:34.037031 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.037006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5l8m9" event={"ID":"e0aaadf1-3ea1-463d-b305-7fa7999dcfaf","Type":"ContainerStarted","Data":"c3892c8e358c0de0dc830d56f51dacb9d7939586694b370951868c49f4c114e1"}
Sep 14 10:25:34.038539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.038515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vwdsr" event={"ID":"b41dccc1-059f-48f1-8a80-453ce7108d65","Type":"ContainerStarted","Data":"c87eea692a655299a0fa00019d9cd745b734ba3aa1ca25514609631284ba130d"}
Sep 14 10:25:34.041174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.041134    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rvl4z" podStartSLOduration=7.163757903 podStartE2EDuration="22.041121674s" totalImagesPullingTime="14.877363771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:34.041112318 +0000 UTC m=+22.596300244" watchObservedRunningTime="2026-09-14 10:25:34.041121674 +0000 UTC m=+22.596309601"
Sep 14 10:25:34.041854 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.041825    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" podStartSLOduration=20.041815595 podStartE2EDuration="20.041815595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:16.996857622 +0000 UTC m=+5.552045551" watchObservedRunningTime="2026-09-14 10:25:34.041815595 +0000 UTC m=+22.597003520"
Sep 14 10:25:34.082619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.082589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:34.088499 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.088455    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vwdsr" podStartSLOduration=3.653459932 podStartE2EDuration="22.088439789s" totalImagesPullingTime="18.434979857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:34.088002183 +0000 UTC m=+22.643190111" watchObservedRunningTime="2026-09-14 10:25:34.088439789 +0000 UTC m=+22.643627717"
Sep 14 10:25:34.088747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.088719    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6xzkn" podStartSLOduration=2.638574958 podStartE2EDuration="21.088706414s" totalImagesPullingTime="18.450131456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:34.063628585 +0000 UTC m=+22.618816513" watchObservedRunningTime="2026-09-14 10:25:34.088706414 +0000 UTC m=+22.643894343"
Sep 14 10:25:34.125194 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.125161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hwrdf" podStartSLOduration=2.688425499 podStartE2EDuration="21.125150648s" totalImagesPullingTime="18.436725149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:34.104319925 +0000 UTC m=+22.659507850" watchObservedRunningTime="2026-09-14 10:25:34.125150648 +0000 UTC m=+22.680338572"
Sep 14 10:25:34.153148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.153111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:34.281150 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.281065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:34.661924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.661902    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 14 10:25:34.690881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.690850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:34.848301 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.848050    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T10:25:34.66192051Z","UUID":"d753fba6-9bb8-408c-ab42-18d9a57f2d65","Handler":null,"Name":"","Endpoint":""}
Sep 14 10:25:34.850603 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.850563    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 14 10:25:34.850719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.850617    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 14 10:25:34.912746 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.912714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:34.913168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:34.912714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:34.913168 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:34.912829    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:34.913168 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:34.912880    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:35.042596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.042525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xk5c" event={"ID":"a59facbf-7d67-430f-a4c9-38456ddf76d3","Type":"ContainerStarted","Data":"bd238c6b95bc397a746c7309a493543d0dd25b991246766808da246203ee924a"}
Sep 14 10:25:35.044984 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.044953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b" event={"ID":"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be","Type":"ContainerStarted","Data":"f4f2bfeb3413df9321c6fc4f43d3adf760da41a896427435ce8aaa90bff53603"}
Sep 14 10:25:35.060459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.060406    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5l8m9" podStartSLOduration=3.6303474529999997 podStartE2EDuration="22.06038481s" totalImagesPullingTime="18.430037357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:34.13838295 +0000 UTC m=+22.693570887" watchObservedRunningTime="2026-09-14 10:25:35.06038481 +0000 UTC m=+23.615572737"
Sep 14 10:25:35.060601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.060507    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4xk5c" podStartSLOduration=4.698395204 podStartE2EDuration="23.060499165s" totalImagesPullingTime="18.362103961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:35.059665036 +0000 UTC m=+23.614852965" watchObservedRunningTime="2026-09-14 10:25:35.060499165 +0000 UTC m=+23.615687092"
Sep 14 10:25:35.502842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.502808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:35.876404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.876366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:35.878355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.878337    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:35.878833 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.878814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:35.916803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:35.916768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:35.917092 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:35.916895    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:36.051612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.051515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"1ed7fd514e33b9c3ddf55b2d906e68c0b1064d6edfb27a4a0711404c480e38b3"}
Sep 14 10:25:36.054078 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.054050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b" event={"ID":"9c2fdbe1-ea25-4621-9ac4-da1ac70f60be","Type":"ContainerStarted","Data":"509ac95864799d099d1bdbfb409adf3a2749d297605182af8987d506862538bb"}
Sep 14 10:25:36.054546 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.054518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:36.054847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.054832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rvl4z"
Sep 14 10:25:36.082430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.082405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:36.091327 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.091279    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wm96b" podStartSLOduration=2.407417796 podStartE2EDuration="23.091262743s" totalImagesPullingTime="20.683844947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:36.09055834 +0000 UTC m=+24.645746265" watchObservedRunningTime="2026-09-14 10:25:36.091262743 +0000 UTC m=+24.646450670"
Sep 14 10:25:36.280240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.280207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:36.510857 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.510822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:36.912595 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.912535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:36.912595 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:36.912556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:36.912816 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:36.912661    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:36.912816 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:36.912769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:37.598479 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:37.598441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:37.699743 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:37.699688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:37.796918 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:37.796884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:37.897741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:37.897710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:37.916305 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:37.916284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:37.916453 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:37.916388    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:37.997431 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:37.997397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:38.099163 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.099134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:38.188836 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.188748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:38.273834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.273651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:38.375207 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.375044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:38.474558 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.474524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:38.576751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.576714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:38.912327 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.912300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:38.912818 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:38.912393    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:38.912818 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.912450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:38.912818 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:38.912531    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:38.998540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:38.998511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:39.061874 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.061852    2563 generic.go:356] "Generic (PLEG): container finished" podID="a18d0c66-674c-45fc-8fbe-d283baeeb203" containerID="906b04d22f7ea9201262bac690e1fb7a851dc9beeb864359b7161ffd2afc5d28" exitCode=0
Sep 14 10:25:39.061956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.061927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerDied","Data":"906b04d22f7ea9201262bac690e1fb7a851dc9beeb864359b7161ffd2afc5d28"}
Sep 14 10:25:39.065679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.065655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" event={"ID":"462c8353-8c49-416b-9cf4-3a5c680574d6","Type":"ContainerStarted","Data":"902b98f3d66e90a7451fc731b25583f4dce0a3283c8b066f96c0d8e01523e8e2"}
Sep 14 10:25:39.065870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.065851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:39.065930 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.065914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:39.080925 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.080908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:39.092712 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.092695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:39.114472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.114436    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j556p" podStartSLOduration=7.629185678 podStartE2EDuration="26.114425127s" totalImagesPullingTime="18.485239449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:39.113682766 +0000 UTC m=+27.668870692" watchObservedRunningTime="2026-09-14 10:25:39.114425127 +0000 UTC m=+27.669613052"
Sep 14 10:25:39.395473 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.395444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:39.548894 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.548869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:39.912470 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:39.912446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:39.912991 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:39.912587    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:40.068282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:40.068258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:40.083345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:40.083322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:25:40.601203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:40.601132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:40.666464 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:40.666434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:40.912784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:40.912758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:40.913240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:40.912759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:40.913240 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:40.912863    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:40.913240 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:40.912912    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:41.071954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:41.071923    2563 generic.go:356] "Generic (PLEG): container finished" podID="a18d0c66-674c-45fc-8fbe-d283baeeb203" containerID="d249e5de944b4e7000ed6cc0aa31ca5b83c9b792d8f00c35836fb6e2a6d98285" exitCode=0
Sep 14 10:25:41.072118 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:41.072011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerDied","Data":"d249e5de944b4e7000ed6cc0aa31ca5b83c9b792d8f00c35836fb6e2a6d98285"}
Sep 14 10:25:41.913520 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:41.913498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:41.913811 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:41.913606    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:42.076347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:42.076318    2563 generic.go:356] "Generic (PLEG): container finished" podID="a18d0c66-674c-45fc-8fbe-d283baeeb203" containerID="76f3c81ef574349e8508bd9a2f32042fe2802f1f5632c06c397584746e91f860" exitCode=0
Sep 14 10:25:42.076445 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:42.076407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerDied","Data":"76f3c81ef574349e8508bd9a2f32042fe2802f1f5632c06c397584746e91f860"}
Sep 14 10:25:42.912655 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:42.912619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:42.912822 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:42.912635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:42.912822 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:42.912727    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:42.912822 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:42.912787    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:43.220362 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:43.220278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:43.314322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:43.314293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:43.912669 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:43.912635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:43.912857 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:43.912779    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:44.912643 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:44.912608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:44.913058 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:44.912609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:44.913058 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:44.912742    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:44.913058 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:44.912818    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:45.549970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:45.549929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:45.549970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:45.549971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:45.550185 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.550076    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:45.550185 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.550094    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:45.550185 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.550131    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs podName:c857632f-81d2-46e9-b1fb-f6921245a682 nodeName:}" failed. No retries permitted until 2026-09-14 10:26:17.550115797 +0000 UTC m=+66.105303704 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs") pod "network-metrics-daemon-pq8l7" (UID: "c857632f-81d2-46e9-b1fb-f6921245a682") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 14 10:25:45.550185 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.550159    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret podName:7b35dd04-3950-42a5-9755-03cb4f5059b6 nodeName:}" failed. No retries permitted until 2026-09-14 10:26:17.550141068 +0000 UTC m=+66.105328995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret") pod "global-pull-secret-syncer-tbshl" (UID: "7b35dd04-3950-42a5-9755-03cb4f5059b6") : object "kube-system"/"original-pull-secret" not registered
Sep 14 10:25:45.650625 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:45.650590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:45.650788 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.650771    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 14 10:25:45.650832 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.650792    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 14 10:25:45.650832 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.650802    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5h2nx for pod openshift-network-diagnostics/network-check-target-mx88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:45.650888 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.650855    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx podName:7144581f-3302-4520-aade-92b01e5f7058 nodeName:}" failed. No retries permitted until 2026-09-14 10:26:17.650841078 +0000 UTC m=+66.206028982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5h2nx" (UniqueName: "kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx") pod "network-check-target-mx88d" (UID: "7144581f-3302-4520-aade-92b01e5f7058") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 14 10:25:45.912741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:45.912706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:45.913180 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:45.912858    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:46.912095 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:46.912068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:46.912285 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:46.912163    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:46.912285 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:46.912219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:46.912354 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:46.912300    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:47.023795 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:47.023771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:47.089199 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:47.089175    2563 generic.go:356] "Generic (PLEG): container finished" podID="a18d0c66-674c-45fc-8fbe-d283baeeb203" containerID="39e9fa9eb81f94d0c8bfb4ce866aabcc838b9c57053f1eb0b11991300ba95f7f" exitCode=0
Sep 14 10:25:47.089287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:47.089224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerDied","Data":"39e9fa9eb81f94d0c8bfb4ce866aabcc838b9c57053f1eb0b11991300ba95f7f"}
Sep 14 10:25:47.124559 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:47.124534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:47.912581 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:47.912555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:47.912735 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:47.912668    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:48.094465 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.094433    2563 generic.go:356] "Generic (PLEG): container finished" podID="a18d0c66-674c-45fc-8fbe-d283baeeb203" containerID="45d6591605ec60b1d72d8d6ac0511c2324b1a6b7c6ff39c7c53e12770056ce51" exitCode=0
Sep 14 10:25:48.094806 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.094489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerDied","Data":"45d6591605ec60b1d72d8d6ac0511c2324b1a6b7c6ff39c7c53e12770056ce51"}
Sep 14 10:25:48.133473 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.133444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:48.248982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.248957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:48.493812 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.493742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:48.686682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.686645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:48.785266 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.785184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:48.912896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.912861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:48.912896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.912880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:48.913077 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:48.912957    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:48.913115 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:48.913092    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:48.995487 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:48.995455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:49.096548 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.096451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:49.101759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.101722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" event={"ID":"a18d0c66-674c-45fc-8fbe-d283baeeb203","Type":"ContainerStarted","Data":"c251feb24e9690b3fc80ca823ae58e12a18cc56c6c53d06e0b5be96575397756"}
Sep 14 10:25:49.126993 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.126895    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f4hh7" podStartSLOduration=7.495422825 podStartE2EDuration="36.1268743s" totalImagesPullingTime="28.631451475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:49.12677845 +0000 UTC m=+37.681966377" watchObservedRunningTime="2026-09-14 10:25:49.1268743 +0000 UTC m=+37.682062229"
Sep 14 10:25:49.191520 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.191489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:49.282982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.282949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:49.362517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.362177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mx88d"]
Sep 14 10:25:49.362517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.362278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:49.362517 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:49.362354    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:49.366780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.366754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pq8l7"]
Sep 14 10:25:49.366904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.366855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:49.366980 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:49.366959    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:49.368654 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.368352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tbshl"]
Sep 14 10:25:49.368826 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.368805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:49.368942 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:49.368900    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:49.377901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.377876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:49.460646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.460617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:49.517908 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:49.517883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:50.912245 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:50.912214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:50.912610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:50.912251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:50.912610 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:50.912298    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:50.912610 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:50.912356    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:50.912610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:50.912397    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:50.912610 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:50.912447    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:51.440146 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:51.440112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:51.531422 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:51.531391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:52.912197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:52.912032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:52.912643 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:52.912033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:52.912643 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:52.912264    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq8l7" podUID="c857632f-81d2-46e9-b1fb-f6921245a682"
Sep 14 10:25:52.912643 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:52.912356    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mx88d" podUID="7144581f-3302-4520-aade-92b01e5f7058"
Sep 14 10:25:52.912643 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:52.912032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:52.912643 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:25:52.912456    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tbshl" podUID="7b35dd04-3950-42a5-9755-03cb4f5059b6"
Sep 14 10:25:53.354134 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:53.354103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:53.458797 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:53.458768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:53.558972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:53.558894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:53.674736 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:53.674699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:54.194062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.194036    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeReady"
Sep 14 10:25:54.194420 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.194169    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 14 10:25:54.260330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.260301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hmlhh"]
Sep 14 10:25:54.284548 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.284520    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sl6hx"]
Sep 14 10:25:54.284685 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.284655    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.287015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.286994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k6n66\""
Sep 14 10:25:54.287158 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.287000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 14 10:25:54.287158 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.287060    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 14 10:25:54.287347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.287329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 14 10:25:54.304747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.304727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hmlhh"]
Sep 14 10:25:54.304845 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.304757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sl6hx"]
Sep 14 10:25:54.304845 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.304784    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9fqmh"]
Sep 14 10:25:54.304911 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.304844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.307655 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.307639    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 14 10:25:54.308176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.308161    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 14 10:25:54.308258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.308237    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 14 10:25:54.308314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.308244    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 14 10:25:54.308314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.308304    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-msf85\""
Sep 14 10:25:54.324690 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.324669    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fqmh"]
Sep 14 10:25:54.324752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.324710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.327644 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.327623    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dh7mm\""
Sep 14 10:25:54.327737 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.327724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 14 10:25:54.327814 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.327775    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 14 10:25:54.417331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-crio-socket\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.417408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.417408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/029e515a-26fc-4f35-b198-36597938ede6-cert\") pod \"ingress-canary-hmlhh\" (UID: \"029e515a-26fc-4f35-b198-36597938ede6\") " pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.417408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmnxk\" (UniqueName: \"kubernetes.io/projected/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-kube-api-access-xmnxk\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.417408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-config-volume\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.417534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gd62\" (UniqueName: \"kubernetes.io/projected/029e515a-26fc-4f35-b198-36597938ede6-kube-api-access-6gd62\") pod \"ingress-canary-hmlhh\" (UID: \"029e515a-26fc-4f35-b198-36597938ede6\") " pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.417534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-metrics-tls\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.417534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-tmp-dir\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.417534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztlwf\" (UniqueName: \"kubernetes.io/projected/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-kube-api-access-ztlwf\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.417534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-data-volume\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.417694 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.417545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.428286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.428249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:54.518419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztlwf\" (UniqueName: \"kubernetes.io/projected/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-kube-api-access-ztlwf\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.518419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-data-volume\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.518419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.518691 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-crio-socket\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.518691 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.518691 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/029e515a-26fc-4f35-b198-36597938ede6-cert\") pod \"ingress-canary-hmlhh\" (UID: \"029e515a-26fc-4f35-b198-36597938ede6\") " pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.518691 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmnxk\" (UniqueName: \"kubernetes.io/projected/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-kube-api-access-xmnxk\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.518898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-crio-socket\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.518898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-config-volume\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.518898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gd62\" (UniqueName: \"kubernetes.io/projected/029e515a-26fc-4f35-b198-36597938ede6-kube-api-access-6gd62\") pod \"ingress-canary-hmlhh\" (UID: \"029e515a-26fc-4f35-b198-36597938ede6\") " pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.518898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-metrics-tls\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.518898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-data-volume\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.518898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.518803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-tmp-dir\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.519175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.519113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.519175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.519132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-tmp-dir\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.519265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.519248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-config-volume\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.522682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.522656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-metrics-tls\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.522759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.522681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.522759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.522712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/029e515a-26fc-4f35-b198-36597938ede6-cert\") pod \"ingress-canary-hmlhh\" (UID: \"029e515a-26fc-4f35-b198-36597938ede6\") " pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.529903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.529881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmnxk\" (UniqueName: \"kubernetes.io/projected/3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1-kube-api-access-xmnxk\") pod \"insights-runtime-extractor-sl6hx\" (UID: \"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1\") " pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.529997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.529970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gd62\" (UniqueName: \"kubernetes.io/projected/029e515a-26fc-4f35-b198-36597938ede6-kube-api-access-6gd62\") pod \"ingress-canary-hmlhh\" (UID: \"029e515a-26fc-4f35-b198-36597938ede6\") " pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.531267 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.531248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztlwf\" (UniqueName: \"kubernetes.io/projected/329e2c19-8fd6-4b7c-8d99-3b8b9a336f41-kube-api-access-ztlwf\") pod \"dns-default-9fqmh\" (UID: \"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41\") " pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.594227 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.594204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hmlhh"
Sep 14 10:25:54.615399 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.615381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sl6hx"
Sep 14 10:25:54.631847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.631821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:54.912785 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.912755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:25:54.913014 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.912755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:25:54.913431 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.912768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:25:54.917039 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.917014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 10:25:54.917161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.917047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 10:25:54.917161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.917081    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 10:25:54.917161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.917128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dsz5x\""
Sep 14 10:25:54.917385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.917369    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 10:25:54.917537 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.917491    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjh76\""
Sep 14 10:25:54.935561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.935539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hmlhh"]
Sep 14 10:25:54.938261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.938237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sl6hx"]
Sep 14 10:25:54.940302 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:54.940279    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029e515a_26fc_4f35_b198_36597938ede6.slice/crio-7f9303a777857f62fdbd3e490bf650e43a583a289ee3fc85ef42e0057b4842a3 WatchSource:0}: Error finding container 7f9303a777857f62fdbd3e490bf650e43a583a289ee3fc85ef42e0057b4842a3: Status 404 returned error can't find the container with id 7f9303a777857f62fdbd3e490bf650e43a583a289ee3fc85ef42e0057b4842a3
Sep 14 10:25:54.941761 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:54.941742    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3001c318_bbe2_4f20_bc2a_1b7a84a5b0e1.slice/crio-f25f101ee0c68ddb235a24311096b173dff94620c6da8e2a60d508f02ef9e45d WatchSource:0}: Error finding container f25f101ee0c68ddb235a24311096b173dff94620c6da8e2a60d508f02ef9e45d: Status 404 returned error can't find the container with id f25f101ee0c68ddb235a24311096b173dff94620c6da8e2a60d508f02ef9e45d
Sep 14 10:25:54.947075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:54.947019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fqmh"]
Sep 14 10:25:54.949953 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:54.949937    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329e2c19_8fd6_4b7c_8d99_3b8b9a336f41.slice/crio-c56aa31ee6dce5a4e8ba34f4fad60c9caa40375806b9150ea9a368cbca17d75e WatchSource:0}: Error finding container c56aa31ee6dce5a4e8ba34f4fad60c9caa40375806b9150ea9a368cbca17d75e: Status 404 returned error can't find the container with id c56aa31ee6dce5a4e8ba34f4fad60c9caa40375806b9150ea9a368cbca17d75e
Sep 14 10:25:55.115882 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:55.115703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fqmh" event={"ID":"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41","Type":"ContainerStarted","Data":"c56aa31ee6dce5a4e8ba34f4fad60c9caa40375806b9150ea9a368cbca17d75e"}
Sep 14 10:25:55.117201 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:55.117171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sl6hx" event={"ID":"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1","Type":"ContainerStarted","Data":"55ce9180f8e74ab5f895bd2eaa695cf8d6dddc7bc521dd7fefb1d4caa5d8cda2"}
Sep 14 10:25:55.117325 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:55.117223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sl6hx" event={"ID":"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1","Type":"ContainerStarted","Data":"f25f101ee0c68ddb235a24311096b173dff94620c6da8e2a60d508f02ef9e45d"}
Sep 14 10:25:55.118257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:55.118237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hmlhh" event={"ID":"029e515a-26fc-4f35-b198-36597938ede6","Type":"ContainerStarted","Data":"7f9303a777857f62fdbd3e490bf650e43a583a289ee3fc85ef42e0057b4842a3"}
Sep 14 10:25:56.773737 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:56.773700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:56.853796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:56.853766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:57.126328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:57.126298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sl6hx" event={"ID":"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1","Type":"ContainerStarted","Data":"b5784e5337199a24f6098e10fed70bfb7eab514d4dbcd8a515c2476f3722b7c6"}
Sep 14 10:25:57.407929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:57.407842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:57.478030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:57.477997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:58.011711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.011678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:58.111134 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.111107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:58.131943 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.131496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hmlhh" event={"ID":"029e515a-26fc-4f35-b198-36597938ede6","Type":"ContainerStarted","Data":"45c5c0f17b9c4caaf2417ff04d4d949c0978aac68987b0169335cde4b5aff622"}
Sep 14 10:25:58.133677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.133649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fqmh" event={"ID":"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41","Type":"ContainerStarted","Data":"3c27ac44482aa9c3339cc1d8558478bb15a4f42c66ef89c556c5c70b9ed710a5"}
Sep 14 10:25:58.133792 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.133701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fqmh" event={"ID":"329e2c19-8fd6-4b7c-8d99-3b8b9a336f41","Type":"ContainerStarted","Data":"d24065baa7e82c50c1a52f2f0b06969c2a72d61c7423c381161b5aa9df9b09de"}
Sep 14 10:25:58.133903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.133821    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:25:58.148025 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.147984    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hmlhh" podStartSLOduration=1.272170429 podStartE2EDuration="4.147971925s" totalImagesPullingTime="2.875801496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:58.147323179 +0000 UTC m=+46.702511102" watchObservedRunningTime="2026-09-14 10:25:58.147971925 +0000 UTC m=+46.703159851"
Sep 14 10:25:58.165959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.165913    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9fqmh" podStartSLOduration=1.306361568 podStartE2EDuration="4.16589355s" totalImagesPullingTime="2.859531982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:25:58.165416966 +0000 UTC m=+46.720604895" watchObservedRunningTime="2026-09-14 10:25:58.16589355 +0000 UTC m=+46.721081473"
Sep 14 10:25:58.798478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.798450    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f7f6b9b96-zh5v5"]
Sep 14 10:25:58.814807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.814779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7f6b9b96-zh5v5"]
Sep 14 10:25:58.814969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.814897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:58.817507 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.817467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 14 10:25:58.817843 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.817825    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 14 10:25:58.817956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.817834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 14 10:25:58.817956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.817907    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 14 10:25:58.818085 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.818072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r7wmm\""
Sep 14 10:25:58.818192 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.818149    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 14 10:25:58.818272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.818260    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 14 10:25:58.818381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.818368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 14 10:25:58.822621 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.822602    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 14 10:25:58.952753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.952726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-service-ca\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:58.952885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.952761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-oauth-serving-cert\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:58.952885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.952782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsx6d\" (UniqueName: \"kubernetes.io/projected/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-kube-api-access-bsx6d\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:58.952885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.952799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-trusted-ca-bundle\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:58.952885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.952846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-oauth-config\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:58.953043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.952967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-serving-cert\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:58.953043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:58.953002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-config\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.053467 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.053446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-config\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.053775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.053508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-service-ca\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.053775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.053527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-oauth-serving-cert\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.053775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.053548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsx6d\" (UniqueName: \"kubernetes.io/projected/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-kube-api-access-bsx6d\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.053775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.053584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-trusted-ca-bundle\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.053775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.053623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-oauth-config\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.053775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.053661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-serving-cert\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.054437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.054411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-oauth-serving-cert\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.054523 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.054468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-config\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.054679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.054660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-trusted-ca-bundle\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.058635 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.058501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-oauth-config\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.058733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.058508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-serving-cert\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.063187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.063168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsx6d\" (UniqueName: \"kubernetes.io/projected/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-kube-api-access-bsx6d\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.064824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.064807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-service-ca\") pod \"console-7f7f6b9b96-zh5v5\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") " pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.124298 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.124280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:25:59.212121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.212091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:59.311797 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.311717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:59.403048 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.403015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:25:59.414197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:25:59.414175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f7f6b9b96-zh5v5"]
Sep 14 10:25:59.417548 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:25:59.417527    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddf9a86_b2e7_41ac_9e39_1e8b361e799c.slice/crio-abc37bb88894ec98c6173a78193867827008691b548b00432f543ddf90497bb6 WatchSource:0}: Error finding container abc37bb88894ec98c6173a78193867827008691b548b00432f543ddf90497bb6: Status 404 returned error can't find the container with id abc37bb88894ec98c6173a78193867827008691b548b00432f543ddf90497bb6
Sep 14 10:26:00.141054 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.141014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6b9b96-zh5v5" event={"ID":"cddf9a86-b2e7-41ac-9e39-1e8b361e799c","Type":"ContainerStarted","Data":"abc37bb88894ec98c6173a78193867827008691b548b00432f543ddf90497bb6"}
Sep 14 10:26:00.143518 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.143462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sl6hx" event={"ID":"3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1","Type":"ContainerStarted","Data":"ffdbed6b21af096c535bf8b79715ec902776cde2e05e68264a69f098e10fb9c9"}
Sep 14 10:26:00.163456 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.163281    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sl6hx" podStartSLOduration=2.301994326 podStartE2EDuration="6.163268782s" totalImagesPullingTime="3.861274456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:00.162671924 +0000 UTC m=+48.717859863" watchObservedRunningTime="2026-09-14 10:26:00.163268782 +0000 UTC m=+48.718456706"
Sep 14 10:26:00.554053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.553949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:00.652589 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.652534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:00.720078 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.720049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:00.805161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.805081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:00.905662 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:00.905625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:01.317852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:01.317817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:01.399641 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:01.399599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:01.489931 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:01.489892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:03.157750 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.157712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6b9b96-zh5v5" event={"ID":"cddf9a86-b2e7-41ac-9e39-1e8b361e799c","Type":"ContainerStarted","Data":"c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a"}
Sep 14 10:26:03.177658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.177615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f7f6b9b96-zh5v5" podStartSLOduration=2.299864468 podStartE2EDuration="5.177603552s" totalImagesPullingTime="2.877739084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:03.1767539 +0000 UTC m=+51.731941822" watchObservedRunningTime="2026-09-14 10:26:03.177603552 +0000 UTC m=+51.732791477"
Sep 14 10:26:03.413784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.413708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:03.509960 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.509918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:03.618439 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.618409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:03.658161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.658140    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"]
Sep 14 10:26:03.660209 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.660195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.670004 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.669986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 14 10:26:03.670105 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.670014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 14 10:26:03.670105 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.670021    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 14 10:26:03.670287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.670271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 14 10:26:03.670470 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.670457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jnknn\""
Sep 14 10:26:03.670545 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.670457    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 14 10:26:03.677548 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.677530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"]
Sep 14 10:26:03.688197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.688175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.688279 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.688243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.688320 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.688285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.688354 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.688318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fpc\" (UniqueName: \"kubernetes.io/projected/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-kube-api-access-j5fpc\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.690190 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.690173    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"]
Sep 14 10:26:03.692112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.692098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.699356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.699324    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 14 10:26:03.699555 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.699535    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 14 10:26:03.699650 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.699632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pbjtr\""
Sep 14 10:26:03.700123 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.700109    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 14 10:26:03.708341 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.708323    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"]
Sep 14 10:26:03.711427 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.711403    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l2tft"]
Sep 14 10:26:03.713234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.713218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.716200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.716185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 14 10:26:03.716287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.716209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 14 10:26:03.716287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.716212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c4k9f\""
Sep 14 10:26:03.716708 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.716693    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 14 10:26:03.729732 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.729697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:03.789137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-textfile\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.789238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d214b683-dc68-4867-905b-91d1ef5b86e4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.789371 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-wtmp\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789371 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d214b683-dc68-4867-905b-91d1ef5b86e4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.789371 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.789517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fpc\" (UniqueName: \"kubernetes.io/projected/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-kube-api-access-j5fpc\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.789517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt72k\" (UniqueName: \"kubernetes.io/projected/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-api-access-xt72k\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.789517 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:03.789460    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 14 10:26:03.789517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789517 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:03.789501    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-tls podName:e4f99e74-1d8a-4cbf-a252-351efc1a4d01 nodeName:}" failed. No retries permitted until 2026-09-14 10:26:04.289487796 +0000 UTC m=+52.844675701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-plvcb" (UID: "e4f99e74-1d8a-4cbf-a252-351efc1a4d01") : secret "openshift-state-metrics-tls" not found
Sep 14 10:26:03.789517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90927451-7697-40f0-8240-f20e21ad38a3-metrics-client-ca\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.789763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-root\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmf7\" (UniqueName: \"kubernetes.io/projected/90927451-7697-40f0-8240-f20e21ad38a3-kube-api-access-7zmf7\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.789763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.789906 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-sys\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789906 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-tls\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.789906 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.789817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.790361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.790336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.793400 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.793379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.805815 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.805790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fpc\" (UniqueName: \"kubernetes.io/projected/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-kube-api-access-j5fpc\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:03.827970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.827946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:03.890223 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90927451-7697-40f0-8240-f20e21ad38a3-metrics-client-ca\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.890314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-root\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmf7\" (UniqueName: \"kubernetes.io/projected/90927451-7697-40f0-8240-f20e21ad38a3-kube-api-access-7zmf7\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.890466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-root\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-sys\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-tls\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-textfile\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-sys\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d214b683-dc68-4867-905b-91d1ef5b86e4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:03.890553    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:03.890641    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-tls podName:90927451-7697-40f0-8240-f20e21ad38a3 nodeName:}" failed. No retries permitted until 2026-09-14 10:26:04.39062308 +0000 UTC m=+52.945811013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-tls") pod "node-exporter-l2tft" (UID: "90927451-7697-40f0-8240-f20e21ad38a3") : secret "node-exporter-tls" not found
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-wtmp\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-wtmp\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d214b683-dc68-4867-905b-91d1ef5b86e4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt72k\" (UniqueName: \"kubernetes.io/projected/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-api-access-xt72k\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90927451-7697-40f0-8240-f20e21ad38a3-metrics-client-ca\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-textfile\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d214b683-dc68-4867-905b-91d1ef5b86e4-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.890884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.890808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.891411 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.891036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.891411 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.891288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.891411 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.891363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d214b683-dc68-4867-905b-91d1ef5b86e4-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.892831 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.892814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.893430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.893412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.893541 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.893521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.909918 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.909898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmf7\" (UniqueName: \"kubernetes.io/projected/90927451-7697-40f0-8240-f20e21ad38a3-kube-api-access-7zmf7\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:03.910727 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.910710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt72k\" (UniqueName: \"kubernetes.io/projected/d214b683-dc68-4867-905b-91d1ef5b86e4-kube-api-access-xt72k\") pod \"kube-state-metrics-59fbdd77d-fxb96\" (UID: \"d214b683-dc68-4867-905b-91d1ef5b86e4\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:03.950723 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:03.950645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:04.000071 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.000048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"
Sep 14 10:26:04.049700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.049657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:04.149802 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.149754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:04.275594 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.275512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96"]
Sep 14 10:26:04.278931 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:04.278905    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd214b683_dc68_4867_905b_91d1ef5b86e4.slice/crio-0f43e6b582ab6b69989a945d8e449affe3fd8b3f176ec64409345001311794db WatchSource:0}: Error finding container 0f43e6b582ab6b69989a945d8e449affe3fd8b3f176ec64409345001311794db: Status 404 returned error can't find the container with id 0f43e6b582ab6b69989a945d8e449affe3fd8b3f176ec64409345001311794db
Sep 14 10:26:04.294787 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.294768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:04.297741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.297719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4f99e74-1d8a-4cbf-a252-351efc1a4d01-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-plvcb\" (UID: \"e4f99e74-1d8a-4cbf-a252-351efc1a4d01\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:04.305549 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.305532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:04.395544 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.395521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-tls\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:04.397718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.397702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90927451-7697-40f0-8240-f20e21ad38a3-node-exporter-tls\") pod \"node-exporter-l2tft\" (UID: \"90927451-7697-40f0-8240-f20e21ad38a3\") " pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:04.406246 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.406209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:04.567976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.567922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"
Sep 14 10:26:04.626051 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.625787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2tft"
Sep 14 10:26:04.635062 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:04.635022    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90927451_7697_40f0_8240_f20e21ad38a3.slice/crio-bb7f4cdc4d9a7186f1a8183729b7a1751e70f64b731212255f2d25c17aabdc58 WatchSource:0}: Error finding container bb7f4cdc4d9a7186f1a8183729b7a1751e70f64b731212255f2d25c17aabdc58: Status 404 returned error can't find the container with id bb7f4cdc4d9a7186f1a8183729b7a1751e70f64b731212255f2d25c17aabdc58
Sep 14 10:26:04.673800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.673768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:04.782764 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.782736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:04.789674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.789650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:04.792598 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.792560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.795245 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.795215    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 14 10:26:04.795364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.795284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 14 10:26:04.795364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.795318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 14 10:26:04.795482 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.795420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 14 10:26:04.795639 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.795623    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 14 10:26:04.795740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.795721    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 14 10:26:04.796004 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.795989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 14 10:26:04.796072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.796031    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 14 10:26:04.796199 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.796183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 14 10:26:04.796503 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.796488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fc2p7\""
Sep 14 10:26:04.798812 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.798790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.798899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.798839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvg4\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-kube-api-access-kgvg4\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.798899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.798867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-web-config\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.798899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.798889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.798916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.798976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.799003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.799034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.799060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-out\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.799086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.799116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.799195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.799250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.799235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.810923 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.810897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:04.839741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.839682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb"]
Sep 14 10:26:04.901007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.900586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.900647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-out\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.900676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.900710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:04.901169    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvg4\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-kube-api-access-kgvg4\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:04.901222    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls podName:1a6d40b5-3603-4453-bf6b-8b8b718346ed nodeName:}" failed. No retries permitted until 2026-09-14 10:26:05.401203607 +0000 UTC m=+53.956391512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed") : secret "alertmanager-main-tls" not found
Sep 14 10:26:04.901259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-web-config\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901721 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:04.901365    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle podName:1a6d40b5-3603-4453-bf6b-8b8b718346ed nodeName:}" failed. No retries permitted until 2026-09-14 10:26:05.401344833 +0000 UTC m=+53.956532760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed") : configmap references non-existent config key: ca-bundle.crt
Sep 14 10:26:04.901721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.901721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.901425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.902518 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.902128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.903438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.903402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-out\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.904381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.904352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.904759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.904739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.905515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.905490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.905630 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.905607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-web-config\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.905953 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.905936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.906167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.906144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.907000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.906977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:04.915237 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:04.915212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvg4\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-kube-api-access-kgvg4\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:05.165933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.165897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2tft" event={"ID":"90927451-7697-40f0-8240-f20e21ad38a3","Type":"ContainerStarted","Data":"bb7f4cdc4d9a7186f1a8183729b7a1751e70f64b731212255f2d25c17aabdc58"}
Sep 14 10:26:05.167963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.167924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb" event={"ID":"e4f99e74-1d8a-4cbf-a252-351efc1a4d01","Type":"ContainerStarted","Data":"f9c987b8442dae8c91cd6a6df86e08456a71efa837229863536bfd67d1969f00"}
Sep 14 10:26:05.168103 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.167993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb" event={"ID":"e4f99e74-1d8a-4cbf-a252-351efc1a4d01","Type":"ContainerStarted","Data":"24a093dce80a40cfd80996ae007bfb3aceb3fe2be953569b24e6be5aaf961800"}
Sep 14 10:26:05.168103 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.168022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb" event={"ID":"e4f99e74-1d8a-4cbf-a252-351efc1a4d01","Type":"ContainerStarted","Data":"ff4ace3adb4f738f608f642cbc47e6a29f93245e62d9055df1dc69bc5c5d78a1"}
Sep 14 10:26:05.169262 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.169226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96" event={"ID":"d214b683-dc68-4867-905b-91d1ef5b86e4","Type":"ContainerStarted","Data":"0f43e6b582ab6b69989a945d8e449affe3fd8b3f176ec64409345001311794db"}
Sep 14 10:26:05.179486 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.179461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:05.406141 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.406104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:05.407463 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.406176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:05.407463 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.407245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:05.409851 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.409822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:05.420692 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.420635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:05.721312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:05.720229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:05.725216 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:05.725186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6d40b5_3603_4453_bf6b_8b8b718346ed.slice/crio-90bd07211a2d855ded2037533acfe6b6701dbae01244ef325487f03af72a3195 WatchSource:0}: Error finding container 90bd07211a2d855ded2037533acfe6b6701dbae01244ef325487f03af72a3195: Status 404 returned error can't find the container with id 90bd07211a2d855ded2037533acfe6b6701dbae01244ef325487f03af72a3195
Sep 14 10:26:06.174924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.174890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96" event={"ID":"d214b683-dc68-4867-905b-91d1ef5b86e4","Type":"ContainerStarted","Data":"40b3032f5443ccdb0948a31e2de01c1e5fa7c7e666ca9d64d9ab7e298493887c"}
Sep 14 10:26:06.175028 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.174964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96" event={"ID":"d214b683-dc68-4867-905b-91d1ef5b86e4","Type":"ContainerStarted","Data":"38ebe01b28a40d6dceff07ffb5180d4bd69a8916beec0098582958f39eb033b5"}
Sep 14 10:26:06.175028 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.174991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96" event={"ID":"d214b683-dc68-4867-905b-91d1ef5b86e4","Type":"ContainerStarted","Data":"2f2bbaec75adf4e955401fbff5b72c6b02b1dd6b060c33fee3479cd5eaf13800"}
Sep 14 10:26:06.176213 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.176189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerStarted","Data":"90bd07211a2d855ded2037533acfe6b6701dbae01244ef325487f03af72a3195"}
Sep 14 10:26:06.177777 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.177753    2563 generic.go:356] "Generic (PLEG): container finished" podID="90927451-7697-40f0-8240-f20e21ad38a3" containerID="de79aa7a733c1d85f3f06e675d3d7d56675c1aac08b377ea1ca07db592fbc6ca" exitCode=0
Sep 14 10:26:06.177880 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.177785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2tft" event={"ID":"90927451-7697-40f0-8240-f20e21ad38a3","Type":"ContainerDied","Data":"de79aa7a733c1d85f3f06e675d3d7d56675c1aac08b377ea1ca07db592fbc6ca"}
Sep 14 10:26:06.198847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.198793    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-fxb96" podStartSLOduration=1.8119171170000001 podStartE2EDuration="3.198773059s" totalImagesPullingTime="1.386855942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:06.196870248 +0000 UTC m=+54.752058176" watchObservedRunningTime="2026-09-14 10:26:06.198773059 +0000 UTC m=+54.753960997"
Sep 14 10:26:06.466918 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.466827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:06.567128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.567098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:06.683225 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.683188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:06.692032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.692009    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"]
Sep 14 10:26:06.694846 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.694829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.699062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.699042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 14 10:26:06.699202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.699045    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 14 10:26:06.699627 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.699601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 14 10:26:06.699741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.699655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-55wfd\""
Sep 14 10:26:06.699905 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.699880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dc8u6ljpt6gvh\""
Sep 14 10:26:06.699998 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.699949    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 14 10:26:06.699998 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.699995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 14 10:26:06.709433 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.709411    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"]
Sep 14 10:26:06.718312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.718396 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.718448 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-grpc-tls\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.718503 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-tls\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.718503 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45892b23-fb26-4714-aafa-7be601377d6d-metrics-client-ca\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.718633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.718633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.718633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.718586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhd5z\" (UniqueName: \"kubernetes.io/projected/45892b23-fb26-4714-aafa-7be601377d6d-kube-api-access-qhd5z\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.772745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.772720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:06.819904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.819881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45892b23-fb26-4714-aafa-7be601377d6d-metrics-client-ca\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.819956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.819927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.819985 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.819958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.820014 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.819983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhd5z\" (UniqueName: \"kubernetes.io/projected/45892b23-fb26-4714-aafa-7be601377d6d-kube-api-access-qhd5z\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.820045 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.820028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.820136 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.820120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.820201 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.820153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-grpc-tls\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.820201 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.820185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-tls\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.820864 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.820801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45892b23-fb26-4714-aafa-7be601377d6d-metrics-client-ca\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.822859 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.822835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.823175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.823154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-grpc-tls\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.823235 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.823170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-tls\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.823235 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.823174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.823328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.823244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.823328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.823272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45892b23-fb26-4714-aafa-7be601377d6d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:06.828338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:06.828320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhd5z\" (UniqueName: \"kubernetes.io/projected/45892b23-fb26-4714-aafa-7be601377d6d-kube-api-access-qhd5z\") pod \"thanos-querier-6c8cf6bc66-2q5hg\" (UID: \"45892b23-fb26-4714-aafa-7be601377d6d\") " pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:07.005725 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.005668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:07.034716 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.034693    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:07.171482 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.171452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:07.182881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.182850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2tft" event={"ID":"90927451-7697-40f0-8240-f20e21ad38a3","Type":"ContainerStarted","Data":"1d89235c5336d1edfbc555e78849dc2ba132bdb8fb5f884240d94bee9eceea30"}
Sep 14 10:26:07.182989 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.182904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2tft" event={"ID":"90927451-7697-40f0-8240-f20e21ad38a3","Type":"ContainerStarted","Data":"153f7ac313dcbe35d0331bab8f52615a60260f263865cd39443d346dc1b40747"}
Sep 14 10:26:07.184899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.184877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb" event={"ID":"e4f99e74-1d8a-4cbf-a252-351efc1a4d01","Type":"ContainerStarted","Data":"ab8edc44199b8a19c333b68d937ee3560e1540a98c5fda34ee58e6e53258663d"}
Sep 14 10:26:07.186272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.186250    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerID="20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181" exitCode=0
Sep 14 10:26:07.186351 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.186331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181"}
Sep 14 10:26:07.229235 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.229200    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l2tft" podStartSLOduration=3.090257854 podStartE2EDuration="4.229186146s" totalImagesPullingTime="1.032099574s" totalInitContainerRuntime="106.828718ms" isStatefulPod=true podCreationTimestamp="2026-09-14 10:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:07.226537304 +0000 UTC m=+55.781725227" watchObservedRunningTime="2026-09-14 10:26:07.229186146 +0000 UTC m=+55.784374073"
Sep 14 10:26:07.266554 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.266469    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-plvcb" podStartSLOduration=3.074536957 podStartE2EDuration="4.266455115s" totalImagesPullingTime="1.191918158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:07.264083442 +0000 UTC m=+55.819271367" watchObservedRunningTime="2026-09-14 10:26:07.266455115 +0000 UTC m=+55.821643042"
Sep 14 10:26:07.306380 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.306235    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"]
Sep 14 10:26:07.308714 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:07.308690    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45892b23_fb26_4714_aafa_7be601377d6d.slice/crio-b5c6f0fcd6b1996b2a34d85e1fcfe02c625e41292f75ff3fb22776f87eefa66d WatchSource:0}: Error finding container b5c6f0fcd6b1996b2a34d85e1fcfe02c625e41292f75ff3fb22776f87eefa66d: Status 404 returned error can't find the container with id b5c6f0fcd6b1996b2a34d85e1fcfe02c625e41292f75ff3fb22776f87eefa66d
Sep 14 10:26:07.425712 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.425652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:07.510558 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.510529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:07.580394 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:07.580330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:08.139289 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.139261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9fqmh"
Sep 14 10:26:08.191979 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.191943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" event={"ID":"45892b23-fb26-4714-aafa-7be601377d6d","Type":"ContainerStarted","Data":"b5c6f0fcd6b1996b2a34d85e1fcfe02c625e41292f75ff3fb22776f87eefa66d"}
Sep 14 10:26:08.326965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.326935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:08.440885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.440803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:08.543424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.543394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:08.968608 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.968546    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"]
Sep 14 10:26:08.988181 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.988155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:08.992499 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.992477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 14 10:26:08.992698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.992655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 14 10:26:08.992776 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.992715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 14 10:26:08.992835 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.992796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g2sh7\""
Sep 14 10:26:08.993438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.993420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 14 10:26:08.993581 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:08.993425    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 14 10:26:09.006497 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.006471    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 14 10:26:09.018067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.018043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"]
Sep 14 10:26:09.039760 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.039735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.039889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.039785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-metrics-client-ca\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.039889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.039802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.039995 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.039884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-telemeter-client-tls\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.039995 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.039976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-serving-certs-ca-bundle\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.040099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.040016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-federate-client-tls\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.040099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.040041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhwj\" (UniqueName: \"kubernetes.io/projected/92e7ebcb-9784-4538-98b1-5c599ce428e7-kube-api-access-blhwj\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.040099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.040080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-secret-telemeter-client\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.124765 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.124725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:26:09.124933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.124808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:26:09.130202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.130177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-metrics-client-ca\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-telemeter-client-tls\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-serving-certs-ca-bundle\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-federate-client-tls\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhwj\" (UniqueName: \"kubernetes.io/projected/92e7ebcb-9784-4538-98b1-5c599ce428e7-kube-api-access-blhwj\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-secret-telemeter-client\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.141542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142823 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.142413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-serving-certs-ca-bundle\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142823 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.142619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.142920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.142852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92e7ebcb-9784-4538-98b1-5c599ce428e7-metrics-client-ca\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.145397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.145370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.145513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.145380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-telemeter-client-tls\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.145513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.145454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-federate-client-tls\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.145513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.145507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92e7ebcb-9784-4538-98b1-5c599ce428e7-secret-telemeter-client\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.150116 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.150094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhwj\" (UniqueName: \"kubernetes.io/projected/92e7ebcb-9784-4538-98b1-5c599ce428e7-kube-api-access-blhwj\") pod \"telemeter-client-7c676b8b94-c2zxb\" (UID: \"92e7ebcb-9784-4538-98b1-5c599ce428e7\") " pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.200206 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.200179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:26:09.299211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.299138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"
Sep 14 10:26:09.934975 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:09.934909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c676b8b94-c2zxb"]
Sep 14 10:26:10.003531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.003496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:10.006069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.006043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:10.033137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.033106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.036395 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.036373    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 14 10:26:10.036505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.036419    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6kb6t\""
Sep 14 10:26:10.036593 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.036537    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c2rup7c9luhks\""
Sep 14 10:26:10.041728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.041702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:10.042238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.042220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.042613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.042868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.043058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.043252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.043438    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.043863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.044007    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.044179    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.044537    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.045920    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.046119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.046389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.051513    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.054638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznc2\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-kube-api-access-lznc2\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.055753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.052753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.068223 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:10.068191    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e7ebcb_9784_4538_98b1_5c599ce428e7.slice/crio-7ad6441fe7c65c9b13abc0c2846dbc27ba3f2b18cdea51fd9870d9075367f59a WatchSource:0}: Error finding container 7ad6441fe7c65c9b13abc0c2846dbc27ba3f2b18cdea51fd9870d9075367f59a: Status 404 returned error can't find the container with id 7ad6441fe7c65c9b13abc0c2846dbc27ba3f2b18cdea51fd9870d9075367f59a
Sep 14 10:26:10.130536 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.130512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:10.153542 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznc2\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-kube-api-access-lznc2\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.153933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.153982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.154011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.154041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.154071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.154100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.154132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.154239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.154167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.155210 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.155150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.155510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.155482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.155750 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.155668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.156164 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.156139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.156677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.156655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.156998 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.156967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.157231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.157206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.157645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.157608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.157645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.157636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.157845 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.157803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.158344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.158281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.158344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.158306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.158482 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.158437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.159261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.159237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.159543 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.159524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.159638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.159618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.159820 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.159806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.160370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.160356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.164409 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.164390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznc2\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-kube-api-access-lznc2\") pod \"prometheus-k8s-0\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.202368 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.202343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerStarted","Data":"6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede"}
Sep 14 10:26:10.202475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.202404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerStarted","Data":"da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa"}
Sep 14 10:26:10.202475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.202429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerStarted","Data":"2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3"}
Sep 14 10:26:10.204014 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.203989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" event={"ID":"45892b23-fb26-4714-aafa-7be601377d6d","Type":"ContainerStarted","Data":"17f4cdc9e97de2a7af2bd14db8bffc8fae94e8794f2809595a9f0438f993f4c7"}
Sep 14 10:26:10.205236 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.205213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb" event={"ID":"92e7ebcb-9784-4538-98b1-5c599ce428e7","Type":"ContainerStarted","Data":"7ad6441fe7c65c9b13abc0c2846dbc27ba3f2b18cdea51fd9870d9075367f59a"}
Sep 14 10:26:10.344271 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.344239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:10.633986 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:10.633412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:10.635810 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:10.635778    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cbdaf9_ce7b_4899_964e_ff8f7d74e392.slice/crio-604fe1f2bd9d9c9c7ddf17787c8f81510f3b206510b67bdbdb6739732a411e5b WatchSource:0}: Error finding container 604fe1f2bd9d9c9c7ddf17787c8f81510f3b206510b67bdbdb6739732a411e5b: Status 404 returned error can't find the container with id 604fe1f2bd9d9c9c7ddf17787c8f81510f3b206510b67bdbdb6739732a411e5b
Sep 14 10:26:11.217588 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.216604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerStarted","Data":"96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf"}
Sep 14 10:26:11.217588 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.216708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerStarted","Data":"fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543"}
Sep 14 10:26:11.217588 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.216735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerStarted","Data":"34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00"}
Sep 14 10:26:11.221247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.221051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" event={"ID":"45892b23-fb26-4714-aafa-7be601377d6d","Type":"ContainerStarted","Data":"d4ae5846996f43d4fb6e59d5b707e4d8d54f9ec50d30c2c7552645362016c576"}
Sep 14 10:26:11.221247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.221134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" event={"ID":"45892b23-fb26-4714-aafa-7be601377d6d","Type":"ContainerStarted","Data":"6e68ad3b2a28c93538ab574e9f16c564347aecb6aaaae1ba9bb984128bc79449"}
Sep 14 10:26:11.221247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.221165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" event={"ID":"45892b23-fb26-4714-aafa-7be601377d6d","Type":"ContainerStarted","Data":"33f6434a42b0473bff98854fc30a413e13ec19f8143c7a3c48db57a68b69a22c"}
Sep 14 10:26:11.221247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.221192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" event={"ID":"45892b23-fb26-4714-aafa-7be601377d6d","Type":"ContainerStarted","Data":"38c42b64436a3ebb83b3e1ae8a1acc3fd0945bc5f94d0d8a1f83822ba88e26a3"}
Sep 14 10:26:11.223605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.223136    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerID="1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4" exitCode=0
Sep 14 10:26:11.223605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.223170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4"}
Sep 14 10:26:11.223605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.223225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerStarted","Data":"604fe1f2bd9d9c9c7ddf17787c8f81510f3b206510b67bdbdb6739732a411e5b"}
Sep 14 10:26:11.247382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:11.247326    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.616355848 podStartE2EDuration="7.247309044s" totalImagesPullingTime="3.615821342s" totalInitContainerRuntime="15.131854ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:11.245932025 +0000 UTC m=+59.801119963" watchObservedRunningTime="2026-09-14 10:26:11.247309044 +0000 UTC m=+59.802496972"
Sep 14 10:26:12.089370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.089335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j556p"
Sep 14 10:26:12.230949 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.230913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" event={"ID":"45892b23-fb26-4714-aafa-7be601377d6d","Type":"ContainerStarted","Data":"1a8545a770feb679753fd2d0958bb29ed7dc2ec90bbc0843cc2aabe6113303bc"}
Sep 14 10:26:12.231371 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.231074    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:12.233595 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.233544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb" event={"ID":"92e7ebcb-9784-4538-98b1-5c599ce428e7","Type":"ContainerStarted","Data":"f4e546b3c7b7c8490314b96c3a0b7b5e9b0b4128913da1ea15cdc7afaec54c7a"}
Sep 14 10:26:12.233728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.233627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb" event={"ID":"92e7ebcb-9784-4538-98b1-5c599ce428e7","Type":"ContainerStarted","Data":"9278f4332572a43d123e2d063ac26bbd5f168b981e506d18f288bb1eab024c38"}
Sep 14 10:26:12.233728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.233658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb" event={"ID":"92e7ebcb-9784-4538-98b1-5c599ce428e7","Type":"ContainerStarted","Data":"daf94133ab2ae874a77489860bb3d4ea39d63a1a32e703b95f07842406d9afe0"}
Sep 14 10:26:12.253648 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.253622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:12.265040 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.264997    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg" podStartSLOduration=2.821968241 podStartE2EDuration="6.26498349s" totalImagesPullingTime="3.443015249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:12.263147359 +0000 UTC m=+60.818335284" watchObservedRunningTime="2026-09-14 10:26:12.26498349 +0000 UTC m=+60.820171406"
Sep 14 10:26:12.297046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.296994    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c676b8b94-c2zxb" podStartSLOduration=2.746410746 podStartE2EDuration="4.296978854s" totalImagesPullingTime="1.550568108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:12.295825362 +0000 UTC m=+60.851013287" watchObservedRunningTime="2026-09-14 10:26:12.296978854 +0000 UTC m=+60.852166780"
Sep 14 10:26:12.367428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.367351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:12.600293 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.600257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:12.679963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.679923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:12.757368 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:12.757334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:14.010557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:14.010515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:14.114456 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:14.114414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:14.984968 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:14.984917    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:15.089858 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.089830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:15.247392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.247313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerStarted","Data":"76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8"}
Sep 14 10:26:15.247392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.247386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerStarted","Data":"973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3"}
Sep 14 10:26:15.247552 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.247414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerStarted","Data":"594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67"}
Sep 14 10:26:15.247552 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.247437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerStarted","Data":"c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed"}
Sep 14 10:26:15.247552 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.247468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerStarted","Data":"511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d"}
Sep 14 10:26:15.247552 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.247492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerStarted","Data":"a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b"}
Sep 14 10:26:15.289856 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.289801    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.21367601 podStartE2EDuration="6.289784916s" totalImagesPullingTime="3.048533367s" totalInitContainerRuntime="27.575539ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:15.287528787 +0000 UTC m=+63.842716713" watchObservedRunningTime="2026-09-14 10:26:15.289784916 +0000 UTC m=+63.844972859"
Sep 14 10:26:15.344799 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.344767    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:15.447532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.447488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:15.542781 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:15.542708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:16.444769 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:16.444737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:17.629288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.629256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:26:17.629288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.629290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:26:17.638579 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.638549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 14 10:26:17.639269 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.639254    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 14 10:26:17.642403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.642384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b35dd04-3950-42a5-9755-03cb4f5059b6-original-pull-secret\") pod \"global-pull-secret-syncer-tbshl\" (UID: \"7b35dd04-3950-42a5-9755-03cb4f5059b6\") " pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:26:17.642496 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.642429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c857632f-81d2-46e9-b1fb-f6921245a682-metrics-certs\") pod \"network-metrics-daemon-pq8l7\" (UID: \"c857632f-81d2-46e9-b1fb-f6921245a682\") " pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:26:17.723756 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.723734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tbshl"
Sep 14 10:26:17.730542 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.730519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:26:17.734280 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.734251    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 14 10:26:17.737123 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.737100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjh76\""
Sep 14 10:26:17.744894 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.744876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq8l7"
Sep 14 10:26:17.746109 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.746092    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 14 10:26:17.754713 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:17.754626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2nx\" (UniqueName: \"kubernetes.io/projected/7144581f-3302-4520-aade-92b01e5f7058-kube-api-access-5h2nx\") pod \"network-check-target-mx88d\" (UID: \"7144581f-3302-4520-aade-92b01e5f7058\") " pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:26:18.028871 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.028849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tbshl"]
Sep 14 10:26:18.030638 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:18.030615    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b35dd04_3950_42a5_9755_03cb4f5059b6.slice/crio-e317c824ccd27a8b7ffefdfadb007a3a7215cacd9537e74b84b5faaa3e1042d7 WatchSource:0}: Error finding container e317c824ccd27a8b7ffefdfadb007a3a7215cacd9537e74b84b5faaa3e1042d7: Status 404 returned error can't find the container with id e317c824ccd27a8b7ffefdfadb007a3a7215cacd9537e74b84b5faaa3e1042d7
Sep 14 10:26:18.034927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.034907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dsz5x\""
Sep 14 10:26:18.036363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.036343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pq8l7"]
Sep 14 10:26:18.038781 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:18.038761    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc857632f_81d2_46e9_b1fb_f6921245a682.slice/crio-e152b5e02914ddcd7b53685d6b802182525565d5b2249c2b33e3e83d98f5e307 WatchSource:0}: Error finding container e152b5e02914ddcd7b53685d6b802182525565d5b2249c2b33e3e83d98f5e307: Status 404 returned error can't find the container with id e152b5e02914ddcd7b53685d6b802182525565d5b2249c2b33e3e83d98f5e307
Sep 14 10:26:18.039610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.039596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:26:18.242856 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.242832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c8cf6bc66-2q5hg"
Sep 14 10:26:18.260012 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.259986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tbshl" event={"ID":"7b35dd04-3950-42a5-9755-03cb4f5059b6","Type":"ContainerStarted","Data":"e317c824ccd27a8b7ffefdfadb007a3a7215cacd9537e74b84b5faaa3e1042d7"}
Sep 14 10:26:18.261180 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.261158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pq8l7" event={"ID":"c857632f-81d2-46e9-b1fb-f6921245a682","Type":"ContainerStarted","Data":"e152b5e02914ddcd7b53685d6b802182525565d5b2249c2b33e3e83d98f5e307"}
Sep 14 10:26:18.335381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:18.335313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mx88d"]
Sep 14 10:26:18.338524 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:18.338499    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7144581f_3302_4520_aade_92b01e5f7058.slice/crio-c545febcab981645582faa928a026d2253ac50de6e7578fc9fb1562cd728c0e1 WatchSource:0}: Error finding container c545febcab981645582faa928a026d2253ac50de6e7578fc9fb1562cd728c0e1: Status 404 returned error can't find the container with id c545febcab981645582faa928a026d2253ac50de6e7578fc9fb1562cd728c0e1
Sep 14 10:26:19.267976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:19.267942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pq8l7" event={"ID":"c857632f-81d2-46e9-b1fb-f6921245a682","Type":"ContainerStarted","Data":"110a9cabe1e5a1ecc918fcecfc9da22c50bbbfd34949a5686058ca165fff2596"}
Sep 14 10:26:19.270432 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:19.270404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mx88d" event={"ID":"7144581f-3302-4520-aade-92b01e5f7058","Type":"ContainerStarted","Data":"c545febcab981645582faa928a026d2253ac50de6e7578fc9fb1562cd728c0e1"}
Sep 14 10:26:19.353735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:19.353475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:19.513141 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:19.513056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:20.278388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:20.278340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pq8l7" event={"ID":"c857632f-81d2-46e9-b1fb-f6921245a682","Type":"ContainerStarted","Data":"ee3bbd809d13d76fb7f925eff433c977b93070b737762539b270705fa6b92678"}
Sep 14 10:26:20.299235 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:20.299179    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pq8l7" podStartSLOduration=66.262966386 podStartE2EDuration="1m7.299160662s" totalImagesPullingTime="1.036194276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:20.297963186 +0000 UTC m=+68.853151180" watchObservedRunningTime="2026-09-14 10:26:20.299160662 +0000 UTC m=+68.854348590"
Sep 14 10:26:20.468397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:20.468348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:20.576781 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:20.576707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:20.683939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:20.683904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:23.294754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.294659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mx88d" event={"ID":"7144581f-3302-4520-aade-92b01e5f7058","Type":"ContainerStarted","Data":"494dd87f070298ad04f71f838d1a415cf726a89768a008bbd8eefcccddec0322"}
Sep 14 10:26:23.295370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.295343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:26:23.297053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.297020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tbshl" event={"ID":"7b35dd04-3950-42a5-9755-03cb4f5059b6","Type":"ContainerStarted","Data":"bc3fdf51ab8585fd63da0cd3685beedf430172fd59762ac2823a80d95c7495fb"}
Sep 14 10:26:23.340303 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.340277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:23.357850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.357789    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tbshl" podStartSLOduration=65.480930795 podStartE2EDuration="1m10.357770357s" totalImagesPullingTime="4.876839562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:23.356145459 +0000 UTC m=+71.911333385" watchObservedRunningTime="2026-09-14 10:26:23.357770357 +0000 UTC m=+71.912958286"
Sep 14 10:26:23.358479 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.358452    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mx88d" podStartSLOduration=65.793224106 podStartE2EDuration="1m10.358442368s" totalImagesPullingTime="4.565218262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:23.325226077 +0000 UTC m=+71.880414004" watchObservedRunningTime="2026-09-14 10:26:23.358442368 +0000 UTC m=+71.913630294"
Sep 14 10:26:23.443470 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.443445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:23.518953 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.518921    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7f6b9b96-zh5v5"]
Sep 14 10:26:23.652788 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.652762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:23.754624 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.754586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:23.854774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:23.854745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:24.076837 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:24.076768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:24.161808 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:24.161782    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:25.443928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:25.443895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:25.545171 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:25.545146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:28.515639 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:28.515605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:36.394263 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:36.394142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:36.464550 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:36.464524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:37.364148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:37.364121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:38.317150 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:38.317117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:38.419564 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:38.419532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:41.337699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:41.337665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:48.538280 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.538221    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f7f6b9b96-zh5v5" podUID="cddf9a86-b2e7-41ac-9e39-1e8b361e799c" containerName="console" containerID="cri-o://c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a" gracePeriod=15
Sep 14 10:26:48.783795 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.783771    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:26:48.878381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-config\") pod \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") "
Sep 14 10:26:48.878502 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-service-ca\") pod \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") "
Sep 14 10:26:48.878502 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878445    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-serving-cert\") pod \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") "
Sep 14 10:26:48.878502 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsx6d\" (UniqueName: \"kubernetes.io/projected/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-kube-api-access-bsx6d\") pod \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") "
Sep 14 10:26:48.878502 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-oauth-serving-cert\") pod \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") "
Sep 14 10:26:48.878710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-trusted-ca-bundle\") pod \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") "
Sep 14 10:26:48.878710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-oauth-config\") pod \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\" (UID: \"cddf9a86-b2e7-41ac-9e39-1e8b361e799c\") "
Sep 14 10:26:48.878914 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-config" (OuterVolumeSpecName: "console-config") pod "cddf9a86-b2e7-41ac-9e39-1e8b361e799c" (UID: "cddf9a86-b2e7-41ac-9e39-1e8b361e799c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:48.878977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cddf9a86-b2e7-41ac-9e39-1e8b361e799c" (UID: "cddf9a86-b2e7-41ac-9e39-1e8b361e799c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:48.879030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-service-ca" (OuterVolumeSpecName: "service-ca") pod "cddf9a86-b2e7-41ac-9e39-1e8b361e799c" (UID: "cddf9a86-b2e7-41ac-9e39-1e8b361e799c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:48.879030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.878998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cddf9a86-b2e7-41ac-9e39-1e8b361e799c" (UID: "cddf9a86-b2e7-41ac-9e39-1e8b361e799c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:48.880935 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.880914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-kube-api-access-bsx6d" (OuterVolumeSpecName: "kube-api-access-bsx6d") pod "cddf9a86-b2e7-41ac-9e39-1e8b361e799c" (UID: "cddf9a86-b2e7-41ac-9e39-1e8b361e799c"). InnerVolumeSpecName "kube-api-access-bsx6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:26:48.881029 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.880930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cddf9a86-b2e7-41ac-9e39-1e8b361e799c" (UID: "cddf9a86-b2e7-41ac-9e39-1e8b361e799c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:48.881029 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.880985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cddf9a86-b2e7-41ac-9e39-1e8b361e799c" (UID: "cddf9a86-b2e7-41ac-9e39-1e8b361e799c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:48.979957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.979927    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-serving-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:48.979957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.979951    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsx6d\" (UniqueName: \"kubernetes.io/projected/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-kube-api-access-bsx6d\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:48.979957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.979960    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-oauth-serving-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:48.980155 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.979969    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-trusted-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:48.980155 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.979977    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-oauth-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:48.980155 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.979985    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-console-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:48.980155 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:48.979993    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cddf9a86-b2e7-41ac-9e39-1e8b361e799c-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:49.295208 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.295120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:49.393559 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.393524    2563 generic.go:356] "Generic (PLEG): container finished" podID="cddf9a86-b2e7-41ac-9e39-1e8b361e799c" containerID="c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a" exitCode=0
Sep 14 10:26:49.393747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.393603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6b9b96-zh5v5" event={"ID":"cddf9a86-b2e7-41ac-9e39-1e8b361e799c","Type":"ContainerDied","Data":"c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a"}
Sep 14 10:26:49.393747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.393611    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f7f6b9b96-zh5v5"
Sep 14 10:26:49.393747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.393672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f7f6b9b96-zh5v5" event={"ID":"cddf9a86-b2e7-41ac-9e39-1e8b361e799c","Type":"ContainerDied","Data":"abc37bb88894ec98c6173a78193867827008691b548b00432f543ddf90497bb6"}
Sep 14 10:26:49.393747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.393693    2563 scope.go:117] "RemoveContainer" containerID="c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a"
Sep 14 10:26:49.402397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.402374    2563 scope.go:117] "RemoveContainer" containerID="c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a"
Sep 14 10:26:49.402746 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:49.402725    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a\": container with ID starting with c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a not found: ID does not exist" containerID="c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a"
Sep 14 10:26:49.402801 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.402754    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a"} err="failed to get container status \"c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a\": rpc error: code = NotFound desc = could not find container \"c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a\": container with ID starting with c7337b3d79d43d76d07663bd9d51f822f73296c8b1db95da527fd8a9b0de104a not found: ID does not exist"
Sep 14 10:26:49.445975 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.445945    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f7f6b9b96-zh5v5"]
Sep 14 10:26:49.451779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.451746    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f7f6b9b96-zh5v5"]
Sep 14 10:26:49.454956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.454930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:49.916642 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:49.916608    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cddf9a86-b2e7-41ac-9e39-1e8b361e799c" path="/var/lib/kubelet/pods/cddf9a86-b2e7-41ac-9e39-1e8b361e799c/volumes"
Sep 14 10:26:50.513723 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:50.513669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:50.636641 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:50.636598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:51.167929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:51.167895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:51.448664 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:51.448526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:53.379788 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.379747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:53.479881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.479847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:26:53.796540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.796460    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:53.797104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.797076    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="alertmanager" containerID="cri-o://2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3" gracePeriod=120
Sep 14 10:26:53.797362 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.797159    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-metric" containerID="cri-o://fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543" gracePeriod=120
Sep 14 10:26:53.797462 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.797378    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-web" containerID="cri-o://6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede" gracePeriod=120
Sep 14 10:26:53.797462 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.797334    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="prom-label-proxy" containerID="cri-o://96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf" gracePeriod=120
Sep 14 10:26:53.797601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.797464    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="config-reloader" containerID="cri-o://da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa" gracePeriod=120
Sep 14 10:26:53.797601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:53.797477    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy" containerID="cri-o://34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00" gracePeriod=120
Sep 14 10:26:54.418189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418152    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerID="96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf" exitCode=0
Sep 14 10:26:54.418189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418181    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerID="fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543" exitCode=0
Sep 14 10:26:54.418189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418189    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerID="34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00" exitCode=0
Sep 14 10:26:54.418189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418195    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerID="da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa" exitCode=0
Sep 14 10:26:54.418659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418201    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerID="2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3" exitCode=0
Sep 14 10:26:54.418659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf"}
Sep 14 10:26:54.418659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543"}
Sep 14 10:26:54.418659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00"}
Sep 14 10:26:54.418659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa"}
Sep 14 10:26:54.418659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:54.418447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3"}
Sep 14 10:26:55.123082 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.123053    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.138956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-main-db\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-web-config\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-volume\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-tls-assets\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139288    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-cluster-tls-config\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgvg4\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-kube-api-access-kgvg4\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-metrics-client-ca\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.139660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.139439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-out\") pod \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\" (UID: \"1a6d40b5-3603-4453-bf6b-8b8b718346ed\") "
Sep 14 10:26:55.140404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.140336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:55.141436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.140621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:26:55.141436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.141193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:55.149897 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.149858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:26:55.150009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.149913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:55.150073 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.150009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:55.150193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.150168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:55.150261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.150120    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-kube-api-access-kgvg4" (OuterVolumeSpecName: "kube-api-access-kgvg4") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "kube-api-access-kgvg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:26:55.150261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.150233    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:55.150967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.150850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-out" (OuterVolumeSpecName: "config-out") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:26:55.151669 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.151641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-volume" (OuterVolumeSpecName: "config-volume") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:55.152796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.152771    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:55.159677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.159653    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-web-config" (OuterVolumeSpecName: "web-config") pod "1a6d40b5-3603-4453-bf6b-8b8b718346ed" (UID: "1a6d40b5-3603-4453-bf6b-8b8b718346ed"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:55.240559 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240525    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240561    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-tls-assets\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240598    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240614    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-main-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240627    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-cluster-tls-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240640    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgvg4\" (UniqueName: \"kubernetes.io/projected/1a6d40b5-3603-4453-bf6b-8b8b718346ed-kube-api-access-kgvg4\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240653    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6d40b5-3603-4453-bf6b-8b8b718346ed-metrics-client-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240666    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-out\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240678    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1a6d40b5-3603-4453-bf6b-8b8b718346ed-alertmanager-main-db\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240694    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240708    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.240719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240720    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-web-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.241027 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.240733    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1a6d40b5-3603-4453-bf6b-8b8b718346ed-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:55.309832 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.309806    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mx88d"
Sep 14 10:26:55.426946 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.426872    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerID="6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede" exitCode=0
Sep 14 10:26:55.426946 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.426917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede"}
Sep 14 10:26:55.427414 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.426971    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.427414 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.426990    2563 scope.go:117] "RemoveContainer" containerID="96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf"
Sep 14 10:26:55.427414 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.426980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1a6d40b5-3603-4453-bf6b-8b8b718346ed","Type":"ContainerDied","Data":"90bd07211a2d855ded2037533acfe6b6701dbae01244ef325487f03af72a3195"}
Sep 14 10:26:55.435429 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.435407    2563 scope.go:117] "RemoveContainer" containerID="fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543"
Sep 14 10:26:55.445046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.445022    2563 scope.go:117] "RemoveContainer" containerID="34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00"
Sep 14 10:26:55.451867 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.451846    2563 scope.go:117] "RemoveContainer" containerID="6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede"
Sep 14 10:26:55.459398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.459379    2563 scope.go:117] "RemoveContainer" containerID="da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa"
Sep 14 10:26:55.460115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.460090    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:55.466582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.466556    2563 scope.go:117] "RemoveContainer" containerID="2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3"
Sep 14 10:26:55.469427 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.469403    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:55.474561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.474539    2563 scope.go:117] "RemoveContainer" containerID="20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181"
Sep 14 10:26:55.481412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.481395    2563 scope.go:117] "RemoveContainer" containerID="96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf"
Sep 14 10:26:55.481691 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:55.481671    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf\": container with ID starting with 96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf not found: ID does not exist" containerID="96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf"
Sep 14 10:26:55.481733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.481700    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf"} err="failed to get container status \"96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf\": rpc error: code = NotFound desc = could not find container \"96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf\": container with ID starting with 96f1f9792417bb5e64f1ba15866f93fb4f20ee343cb4b5f517cb17e445ae63cf not found: ID does not exist"
Sep 14 10:26:55.481733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.481717    2563 scope.go:117] "RemoveContainer" containerID="fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543"
Sep 14 10:26:55.481978 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:55.481958    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543\": container with ID starting with fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543 not found: ID does not exist" containerID="fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543"
Sep 14 10:26:55.482026 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.481984    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543"} err="failed to get container status \"fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543\": rpc error: code = NotFound desc = could not find container \"fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543\": container with ID starting with fb9b931509bbed20a9958c175c06eb8058d357de52ed762464fe1a3887530543 not found: ID does not exist"
Sep 14 10:26:55.482026 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.482000    2563 scope.go:117] "RemoveContainer" containerID="34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00"
Sep 14 10:26:55.482221 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:55.482207    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00\": container with ID starting with 34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00 not found: ID does not exist" containerID="34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00"
Sep 14 10:26:55.482262 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.482224    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00"} err="failed to get container status \"34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00\": rpc error: code = NotFound desc = could not find container \"34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00\": container with ID starting with 34369c8338092a566d0093eb7703faf2b5abb2087d9d7e9f69f039aef1edbe00 not found: ID does not exist"
Sep 14 10:26:55.482262 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.482235    2563 scope.go:117] "RemoveContainer" containerID="6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede"
Sep 14 10:26:55.482475 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:55.482459    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede\": container with ID starting with 6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede not found: ID does not exist" containerID="6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede"
Sep 14 10:26:55.482518 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.482478    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede"} err="failed to get container status \"6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede\": rpc error: code = NotFound desc = could not find container \"6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede\": container with ID starting with 6efc2caef75e45d7ef180d3c95786a4ad8a5624aee1fc3a94e5f2e9db6c59ede not found: ID does not exist"
Sep 14 10:26:55.482518 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.482497    2563 scope.go:117] "RemoveContainer" containerID="da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa"
Sep 14 10:26:55.482768 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:55.482752    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa\": container with ID starting with da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa not found: ID does not exist" containerID="da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa"
Sep 14 10:26:55.482816 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.482771    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa"} err="failed to get container status \"da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa\": rpc error: code = NotFound desc = could not find container \"da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa\": container with ID starting with da0df03465de46e896a7d606ad53abedec3ce3129a252c81e39c082b987232aa not found: ID does not exist"
Sep 14 10:26:55.482816 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.482782    2563 scope.go:117] "RemoveContainer" containerID="2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3"
Sep 14 10:26:55.483043 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:55.483025    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3\": container with ID starting with 2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3 not found: ID does not exist" containerID="2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3"
Sep 14 10:26:55.483080 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.483049    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3"} err="failed to get container status \"2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3\": rpc error: code = NotFound desc = could not find container \"2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3\": container with ID starting with 2d89344e9c1a41949614b7b8609a0b0053ca6edfdf5da5be003a6312c91685b3 not found: ID does not exist"
Sep 14 10:26:55.483080 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.483063    2563 scope.go:117] "RemoveContainer" containerID="20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181"
Sep 14 10:26:55.483274 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:55.483261    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181\": container with ID starting with 20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181 not found: ID does not exist" containerID="20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181"
Sep 14 10:26:55.483310 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.483276    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181"} err="failed to get container status \"20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181\": rpc error: code = NotFound desc = could not find container \"20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181\": container with ID starting with 20e14187aefab777558377b3a1421040815f81b0868b80e7370c79c01f218181 not found: ID does not exist"
Sep 14 10:26:55.536682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.536645    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:55.536980 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.536965    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-web"
Sep 14 10:26:55.537019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.536982    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-web"
Sep 14 10:26:55.537019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.536995    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy"
Sep 14 10:26:55.537019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy"
Sep 14 10:26:55.537019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537009    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="init-config-reloader"
Sep 14 10:26:55.537019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537015    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="init-config-reloader"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537021    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="alertmanager"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537027    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="alertmanager"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537035    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-metric"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537040    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-metric"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537048    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="prom-label-proxy"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537053    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="prom-label-proxy"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537061    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cddf9a86-b2e7-41ac-9e39-1e8b361e799c" containerName="console"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537066    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cddf9a86-b2e7-41ac-9e39-1e8b361e799c" containerName="console"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="config-reloader"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537080    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="config-reloader"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537116    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-metric"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537125    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cddf9a86-b2e7-41ac-9e39-1e8b361e799c" containerName="console"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537131    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="alertmanager"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537136    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537142    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="kube-rbac-proxy-web"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537148    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="config-reloader"
Sep 14 10:26:55.537175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.537154    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" containerName="prom-label-proxy"
Sep 14 10:26:55.540811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.540795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.545531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.545504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 14 10:26:55.545682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.545654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 14 10:26:55.545747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.545656    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 14 10:26:55.545887 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.545853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fc2p7\""
Sep 14 10:26:55.546075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.546046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 14 10:26:55.546196 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.546177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 14 10:26:55.546334 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.546317    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 14 10:26:55.546412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.546353    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 14 10:26:55.546505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.546484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 14 10:26:55.560790 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.560772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 14 10:26:55.587733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.587689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:55.645797 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.645763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8964b79c-02ad-4e1d-a25a-69eefc641450-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.645797 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.645794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-web-config\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.645818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.645864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8964b79c-02ad-4e1d-a25a-69eefc641450-config-out\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.645900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.645936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8964b79c-02ad-4e1d-a25a-69eefc641450-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.645979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w996w\" (UniqueName: \"kubernetes.io/projected/8964b79c-02ad-4e1d-a25a-69eefc641450-kube-api-access-w996w\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.646013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.646041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.646085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8964b79c-02ad-4e1d-a25a-69eefc641450-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.646121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8964b79c-02ad-4e1d-a25a-69eefc641450-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.646145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.646330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.646216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-config-volume\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.746925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8964b79c-02ad-4e1d-a25a-69eefc641450-config-out\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.746958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.746978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8964b79c-02ad-4e1d-a25a-69eefc641450-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.746996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w996w\" (UniqueName: \"kubernetes.io/projected/8964b79c-02ad-4e1d-a25a-69eefc641450-kube-api-access-w996w\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8964b79c-02ad-4e1d-a25a-69eefc641450-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8964b79c-02ad-4e1d-a25a-69eefc641450-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-config-volume\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8964b79c-02ad-4e1d-a25a-69eefc641450-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-web-config\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.747913 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.747891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8964b79c-02ad-4e1d-a25a-69eefc641450-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.748283 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.748250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8964b79c-02ad-4e1d-a25a-69eefc641450-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.748565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.748515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8964b79c-02ad-4e1d-a25a-69eefc641450-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.750010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.749981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8964b79c-02ad-4e1d-a25a-69eefc641450-config-out\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.750299 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.750236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-config-volume\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.750446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.750422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.750446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.750437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.750733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.750711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8964b79c-02ad-4e1d-a25a-69eefc641450-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.750814 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.750740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.750885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.750861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.751132 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.751111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-web-config\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.752137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.752117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8964b79c-02ad-4e1d-a25a-69eefc641450-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.755547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.755531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w996w\" (UniqueName: \"kubernetes.io/projected/8964b79c-02ad-4e1d-a25a-69eefc641450-kube-api-access-w996w\") pod \"alertmanager-main-0\" (UID: \"8964b79c-02ad-4e1d-a25a-69eefc641450\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.849226 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.849204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 14 10:26:55.919817 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:55.919783    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a6d40b5-3603-4453-bf6b-8b8b718346ed" path="/var/lib/kubelet/pods/1a6d40b5-3603-4453-bf6b-8b8b718346ed/volumes"
Sep 14 10:26:56.140221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:56.140193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 14 10:26:56.147253 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:26:56.147230    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8964b79c_02ad_4e1d_a25a_69eefc641450.slice/crio-69a6d830a55efdd743ace0cf64bf06a5d9c06b41c878734d52e398e8008f6985 WatchSource:0}: Error finding container 69a6d830a55efdd743ace0cf64bf06a5d9c06b41c878734d52e398e8008f6985: Status 404 returned error can't find the container with id 69a6d830a55efdd743ace0cf64bf06a5d9c06b41c878734d52e398e8008f6985
Sep 14 10:26:56.432727 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:56.432657    2563 generic.go:356] "Generic (PLEG): container finished" podID="8964b79c-02ad-4e1d-a25a-69eefc641450" containerID="b5580cae12833fd268f7c8d02398691d6490b8dd1a5e5acfa1d1a639674a7442" exitCode=0
Sep 14 10:26:56.433055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:56.432749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerDied","Data":"b5580cae12833fd268f7c8d02398691d6490b8dd1a5e5acfa1d1a639674a7442"}
Sep 14 10:26:56.433055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:56.432856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerStarted","Data":"69a6d830a55efdd743ace0cf64bf06a5d9c06b41c878734d52e398e8008f6985"}
Sep 14 10:26:57.441652 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:57.441618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerStarted","Data":"239808ce50667ed887883c78b5e83eb2c631140d7c9f793e915ad46abb61ba9e"}
Sep 14 10:26:57.442128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:57.441692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerStarted","Data":"0afbf2b5da514cd3580467599e1e430a46c54cdad1aab64fa04ee198aae1c996"}
Sep 14 10:26:57.442128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:57.441738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerStarted","Data":"0f53f0e4ec48d29ab3a22b90860d2e19e3491aea1d8a0477dddf31b7ad922633"}
Sep 14 10:26:57.442128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:57.441780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerStarted","Data":"136293907b8355cdd9bd28aa21aa20213a272947db60a896897d35c55a604b94"}
Sep 14 10:26:57.442128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:57.441824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerStarted","Data":"0c22804ae8c49d7f7e96b43f437c5986ab1cf6fc666ddeb07c4cc24de6bfdb35"}
Sep 14 10:26:57.442128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:57.441861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8964b79c-02ad-4e1d-a25a-69eefc641450","Type":"ContainerStarted","Data":"265cdb16e108cfc982dee1aaed76aeb2ff7ed476e1f99ce1891c87bfd3be868f"}
Sep 14 10:26:57.492224 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:57.492180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.480854053 podStartE2EDuration="2.492160618s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.306565ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:26:57.491646621 +0000 UTC m=+106.046834557" watchObservedRunningTime="2026-09-14 10:26:57.492160618 +0000 UTC m=+106.047348545"
Sep 14 10:26:58.144942 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.144906    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:58.145475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.145236    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="prometheus" containerID="cri-o://a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b" gracePeriod=600
Sep 14 10:26:58.145475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.145269    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="thanos-sidecar" containerID="cri-o://c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed" gracePeriod=600
Sep 14 10:26:58.145475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.145275    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-thanos" containerID="cri-o://76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8" gracePeriod=600
Sep 14 10:26:58.145475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.145336    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="config-reloader" containerID="cri-o://511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d" gracePeriod=600
Sep 14 10:26:58.145475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.145279    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy" containerID="cri-o://973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3" gracePeriod=600
Sep 14 10:26:58.145475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.145289    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-web" containerID="cri-o://594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67" gracePeriod=600
Sep 14 10:26:58.449969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.449886    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerID="76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8" exitCode=0
Sep 14 10:26:58.449969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.449913    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerID="973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3" exitCode=0
Sep 14 10:26:58.449969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.449921    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerID="c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed" exitCode=0
Sep 14 10:26:58.449969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.449928    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerID="511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d" exitCode=0
Sep 14 10:26:58.449969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.449935    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerID="a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b" exitCode=0
Sep 14 10:26:58.449969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.449959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8"}
Sep 14 10:26:58.450702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.450023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3"}
Sep 14 10:26:58.450702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.450048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed"}
Sep 14 10:26:58.450702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.450073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d"}
Sep 14 10:26:58.450702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:58.450095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b"}
Sep 14 10:26:59.379296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.379273    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.456425 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.456361    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerID="594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67" exitCode=0
Sep 14 10:26:59.456751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.456440    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.456751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.456439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67"}
Sep 14 10:26:59.456751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.456605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392","Type":"ContainerDied","Data":"604fe1f2bd9d9c9c7ddf17787c8f81510f3b206510b67bdbdb6739732a411e5b"}
Sep 14 10:26:59.456751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.456653    2563 scope.go:117] "RemoveContainer" containerID="76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8"
Sep 14 10:26:59.464135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.464117    2563 scope.go:117] "RemoveContainer" containerID="973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3"
Sep 14 10:26:59.471144 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.471119    2563 scope.go:117] "RemoveContainer" containerID="594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67"
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-kubelet-serving-ca-bundle\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-1\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475721    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-db\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-trusted-ca-bundle\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config-out\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-thanos-prometheus-http-client-file\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-tls\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-web-config\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.475994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-tls-assets\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-2\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-serving-certs-ca-bundle\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476122    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-0\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.476934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-metrics-client-ca\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-metrics-client-certs\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lznc2\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-kube-api-access-lznc2\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-kube-rbac-proxy\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-grpc-tls\") pod \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\" (UID: \"e0cbdaf9-ce7b-4899-964e-ff8f7d74e392\") "
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476541    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476563    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.478092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.476687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:59.478511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.478302    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:59.480077 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.479987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:59.480398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.480314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:26:59.480490 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.480467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:59.481511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.481488    2563 scope.go:117] "RemoveContainer" containerID="c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed"
Sep 14 10:26:59.482213 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.482178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config" (OuterVolumeSpecName: "config") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.483286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.483064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-kube-api-access-lznc2" (OuterVolumeSpecName: "kube-api-access-lznc2") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "kube-api-access-lznc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:26:59.483286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.483169    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:26:59.483774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.483748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.483909 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.483880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.484452 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.484371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:26:59.484452 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.484405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.484999 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.484799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.484999 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.484861    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.484999 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.484974    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.485900 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.485860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config-out" (OuterVolumeSpecName: "config-out") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:26:59.485989 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.485947    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.496500 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.496467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-web-config" (OuterVolumeSpecName: "web-config") pod "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" (UID: "e0cbdaf9-ce7b-4899-964e-ff8f7d74e392"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:26:59.498809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.498788    2563 scope.go:117] "RemoveContainer" containerID="511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d"
Sep 14 10:26:59.507731 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.507707    2563 scope.go:117] "RemoveContainer" containerID="a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b"
Sep 14 10:26:59.516260 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.516244    2563 scope.go:117] "RemoveContainer" containerID="1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4"
Sep 14 10:26:59.523856 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.523827    2563 scope.go:117] "RemoveContainer" containerID="76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8"
Sep 14 10:26:59.524143 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:59.524123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8\": container with ID starting with 76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8 not found: ID does not exist" containerID="76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8"
Sep 14 10:26:59.524214 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8"} err="failed to get container status \"76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8\": rpc error: code = NotFound desc = could not find container \"76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8\": container with ID starting with 76dd8e755e3133c1f4773f96218f81dcac6ea671bb617b46ffc2754f54a840a8 not found: ID does not exist"
Sep 14 10:26:59.524214 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524169    2563 scope.go:117] "RemoveContainer" containerID="973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3"
Sep 14 10:26:59.524386 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:59.524372    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3\": container with ID starting with 973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3 not found: ID does not exist" containerID="973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3"
Sep 14 10:26:59.524437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524390    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3"} err="failed to get container status \"973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3\": rpc error: code = NotFound desc = could not find container \"973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3\": container with ID starting with 973661962b136785a5fb83b43aa36cbd93415d85e2e1812204bbe1f63d21c1a3 not found: ID does not exist"
Sep 14 10:26:59.524437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524404    2563 scope.go:117] "RemoveContainer" containerID="594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67"
Sep 14 10:26:59.524689 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:59.524672    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67\": container with ID starting with 594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67 not found: ID does not exist" containerID="594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67"
Sep 14 10:26:59.524755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524694    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67"} err="failed to get container status \"594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67\": rpc error: code = NotFound desc = could not find container \"594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67\": container with ID starting with 594a46871161b2fa40684f103900c13244732ffdf5fd5b45eaa6bde9ac434d67 not found: ID does not exist"
Sep 14 10:26:59.524755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524709    2563 scope.go:117] "RemoveContainer" containerID="c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed"
Sep 14 10:26:59.524917 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:59.524901    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed\": container with ID starting with c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed not found: ID does not exist" containerID="c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed"
Sep 14 10:26:59.524969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524921    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed"} err="failed to get container status \"c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed\": rpc error: code = NotFound desc = could not find container \"c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed\": container with ID starting with c2e39a864c1ac2fcdc2aabd4de22cb8014283c2750e81c890a32fa6ab6b7e5ed not found: ID does not exist"
Sep 14 10:26:59.524969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.524934    2563 scope.go:117] "RemoveContainer" containerID="511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d"
Sep 14 10:26:59.525114 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:59.525100    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d\": container with ID starting with 511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d not found: ID does not exist" containerID="511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d"
Sep 14 10:26:59.525170 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.525117    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d"} err="failed to get container status \"511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d\": rpc error: code = NotFound desc = could not find container \"511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d\": container with ID starting with 511ff75894ae56a7a2d9169130a7ceb38869a10830f19f8fc53fd0e10a82eb7d not found: ID does not exist"
Sep 14 10:26:59.525170 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.525130    2563 scope.go:117] "RemoveContainer" containerID="a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b"
Sep 14 10:26:59.525296 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:59.525272    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b\": container with ID starting with a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b not found: ID does not exist" containerID="a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b"
Sep 14 10:26:59.525346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.525299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b"} err="failed to get container status \"a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b\": rpc error: code = NotFound desc = could not find container \"a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b\": container with ID starting with a425d9d581dc454ebc21668bbcd6841aaa487ed153b48427b1b8ee2a653dad1b not found: ID does not exist"
Sep 14 10:26:59.525346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.525312    2563 scope.go:117] "RemoveContainer" containerID="1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4"
Sep 14 10:26:59.525521 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:26:59.525504    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4\": container with ID starting with 1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4 not found: ID does not exist" containerID="1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4"
Sep 14 10:26:59.525627 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.525525    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4"} err="failed to get container status \"1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4\": rpc error: code = NotFound desc = could not find container \"1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4\": container with ID starting with 1308430cecc7f06a579637208e4391a8fb4ab96639c1729e1217f85e6a78d9d4 not found: ID does not exist"
Sep 14 10:26:59.577029 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577000    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-metrics-client-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577029 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577026    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-metrics-client-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577038    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznc2\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-kube-api-access-lznc2\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577046    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-kube-rbac-proxy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577055    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-grpc-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577064    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577072    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577081    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-db\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577089    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-config-out\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577098    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577107    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577114    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-web-config\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577123    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577132    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577140    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-tls-assets\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577149    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577156    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.577198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.577164    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:26:59.788830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.788799    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:59.794218 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.794194    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:59.836578 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836550    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:59.836842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836830    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="config-reloader"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="config-reloader"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836854    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="thanos-sidecar"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836860    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="thanos-sidecar"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836872    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="init-config-reloader"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836877    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="init-config-reloader"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836882    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="prometheus"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836887    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="prometheus"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836892    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy"
Sep 14 10:26:59.836896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836897    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836909    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-web"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836914    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-web"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-thanos"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836925    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-thanos"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836965    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="thanos-sidecar"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836972    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="prometheus"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836980    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836985    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-thanos"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836990    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="kube-rbac-proxy-web"
Sep 14 10:26:59.837149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.836996    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" containerName="config-reloader"
Sep 14 10:26:59.843794 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.843765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.847423 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.847404    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 14 10:26:59.847548 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.847493    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 14 10:26:59.847613 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.847601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 14 10:26:59.847716 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.847701    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 14 10:26:59.847868 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.847852    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 14 10:26:59.848392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.848378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6kb6t\""
Sep 14 10:26:59.848531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.848519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 14 10:26:59.848685 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.848672    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 14 10:26:59.849141 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.849124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 14 10:26:59.849215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.849164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c2rup7c9luhks\""
Sep 14 10:26:59.849215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.849191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 14 10:26:59.849215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.849207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 14 10:26:59.852583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.852554    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 14 10:26:59.853060 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.853045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 14 10:26:59.853134 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.853065    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 14 10:26:59.859243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.859225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 14 10:26:59.863376 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.863360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 14 10:26:59.879697 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac552ec-722c-47a4-a5c7-14b5f986c32e-config-out\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.879697 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.879860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.879860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.879860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.879860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.879860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-web-config\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac552ec-722c-47a4-a5c7-14b5f986c32e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.879999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.880028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-config\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.880063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.880091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.880125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.880145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.880166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltrg\" (UniqueName: \"kubernetes.io/projected/aac552ec-722c-47a4-a5c7-14b5f986c32e-kube-api-access-jltrg\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.880265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.880195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.887405 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.887378    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:26:59.916949 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.916922    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0cbdaf9-ce7b-4899-964e-ff8f7d74e392" path="/var/lib/kubelet/pods/e0cbdaf9-ce7b-4899-964e-ff8f7d74e392/volumes"
Sep 14 10:26:59.981296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-config\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltrg\" (UniqueName: \"kubernetes.io/projected/aac552ec-722c-47a4-a5c7-14b5f986c32e-kube-api-access-jltrg\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac552ec-722c-47a4-a5c7-14b5f986c32e-config-out\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-web-config\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac552ec-722c-47a4-a5c7-14b5f986c32e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.981924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.981827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.982441 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.982038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.982441 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.982066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.983518 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.983322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.983976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.983947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.984999 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.984797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.985220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.985170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.985220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.985178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.985459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.985227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.985459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.985264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.985459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.985286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-config\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.985642 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.985462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.986813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.986770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.986989 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.986968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac552ec-722c-47a4-a5c7-14b5f986c32e-config-out\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.987046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.986991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-web-config\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.987212 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.987195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.987299 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.987280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac552ec-722c-47a4-a5c7-14b5f986c32e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.987387 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.987368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.987767 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.987752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac552ec-722c-47a4-a5c7-14b5f986c32e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.988377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.988362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac552ec-722c-47a4-a5c7-14b5f986c32e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:26:59.994300 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:26:59.994283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltrg\" (UniqueName: \"kubernetes.io/projected/aac552ec-722c-47a4-a5c7-14b5f986c32e-kube-api-access-jltrg\") pod \"prometheus-k8s-0\" (UID: \"aac552ec-722c-47a4-a5c7-14b5f986c32e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:27:00.153601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:00.153558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:27:00.446546 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:00.446479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 14 10:27:00.450776 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:27:00.450752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac552ec_722c_47a4_a5c7_14b5f986c32e.slice/crio-2b794f3ab8165705e45a9e7b76a34b516e23171f53b2a7e9a833b988a9538295 WatchSource:0}: Error finding container 2b794f3ab8165705e45a9e7b76a34b516e23171f53b2a7e9a833b988a9538295: Status 404 returned error can't find the container with id 2b794f3ab8165705e45a9e7b76a34b516e23171f53b2a7e9a833b988a9538295
Sep 14 10:27:00.462963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:00.462935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerStarted","Data":"2b794f3ab8165705e45a9e7b76a34b516e23171f53b2a7e9a833b988a9538295"}
Sep 14 10:27:01.470280 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:01.470201    2563 generic.go:356] "Generic (PLEG): container finished" podID="aac552ec-722c-47a4-a5c7-14b5f986c32e" containerID="81d84d444a8e404268c0036965a8763dcccb84a6c1096137c1afcd4460388b41" exitCode=0
Sep 14 10:27:01.470631 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:01.470294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerDied","Data":"81d84d444a8e404268c0036965a8763dcccb84a6c1096137c1afcd4460388b41"}
Sep 14 10:27:01.829731 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:01.829709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:27:02.022939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.022872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:27:02.160418 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.160390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:27:02.477966 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.477934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerStarted","Data":"c705f74b16c273ee3a8e0f10a3d083292f9cdb7c3398bccf8a88df8074d43512"}
Sep 14 10:27:02.478338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.477996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerStarted","Data":"83372f0b6cf3529cfa9dd50075b4f2dd587dc1ca33258a68c6a2305805f82a1a"}
Sep 14 10:27:02.478338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.478027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerStarted","Data":"14274bb7d9a75a528c934225ea04b2fdb5e6358c7fe503be08346a34bbe0d7d7"}
Sep 14 10:27:02.478338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.478050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerStarted","Data":"d51ee16a19d71424f6ff1da81f89297704cd62bb9ec02909f509b2dd813a9f54"}
Sep 14 10:27:02.478338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.478072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerStarted","Data":"fcaadf56b67c6acea4e5686b05d07cd52d64631e7a8d893c72f04b11d58d4feb"}
Sep 14 10:27:02.478338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.478118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac552ec-722c-47a4-a5c7-14b5f986c32e","Type":"ContainerStarted","Data":"89ae971d05cb34a8f26407ba95dedf66ceebe22b7cdee6882220929c248dbb36"}
Sep 14 10:27:02.516804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:02.516746    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.497502767 podStartE2EDuration="3.516727784s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.225017ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:27:02.51372806 +0000 UTC m=+111.068915987" watchObservedRunningTime="2026-09-14 10:27:02.516727784 +0000 UTC m=+111.071915711"
Sep 14 10:27:05.154332 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:05.154300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:27:40.409850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:40.409806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:27:40.560146 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:27:40.560116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:00.153977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:00.153936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:28:00.168936 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:00.168912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:28:00.714072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:00.714049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 14 10:28:04.838982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:04.838904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:07.283473 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:07.283439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:07.392734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:07.392708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:11.871098 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:11.871065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:12.047167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:12.047131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:13.451651 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:13.451620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:13.804950 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:13.804873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:14.013231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:14.013190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:14.317889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:14.317854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:16.342311 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:16.342282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:16.775279 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:16.775239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:18.114938 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:18.114897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:18.323191 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:18.323161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:22.218646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:22.218593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:22.351133 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:22.351103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:24.254188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:24.254156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:24.918240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:24.918205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:25.115494 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.115465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:25.857030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.856994    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6dgl2"]
Sep 14 10:28:25.862163 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.859858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:25.865465 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.865439    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-g77l4\""
Sep 14 10:28:25.865600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.865466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 14 10:28:25.865600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.865468    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 14 10:28:25.881184 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.881164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6dgl2"]
Sep 14 10:28:25.939728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.939697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6dgl2\" (UID: \"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e\") " pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:25.939899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:25.939736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vtf\" (UniqueName: \"kubernetes.io/projected/86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e-kube-api-access-g7vtf\") pod \"cert-manager-55b4d954d5-6dgl2\" (UID: \"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e\") " pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:26.040832 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.040799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6dgl2\" (UID: \"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e\") " pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:26.040968 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.040836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vtf\" (UniqueName: \"kubernetes.io/projected/86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e-kube-api-access-g7vtf\") pod \"cert-manager-55b4d954d5-6dgl2\" (UID: \"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e\") " pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:26.051185 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.051162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6dgl2\" (UID: \"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e\") " pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:26.051367 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.051351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vtf\" (UniqueName: \"kubernetes.io/projected/86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e-kube-api-access-g7vtf\") pod \"cert-manager-55b4d954d5-6dgl2\" (UID: \"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e\") " pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:26.121979 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.121917    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"]
Sep 14 10:28:26.124088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.124073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.126178 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.126161    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 14 10:28:26.126284 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.126266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9t7g5\""
Sep 14 10:28:26.127257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.127238    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 14 10:28:26.139903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.139882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"]
Sep 14 10:28:26.186108 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.186083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6dgl2"
Sep 14 10:28:26.243164 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.243132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feba8fb8-7800-4069-9c0e-21be998399c2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mgzhb\" (UID: \"feba8fb8-7800-4069-9c0e-21be998399c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.243314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.243288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cnv\" (UniqueName: \"kubernetes.io/projected/feba8fb8-7800-4069-9c0e-21be998399c2-kube-api-access-v9cnv\") pod \"openshift-lws-operator-fd8ccff4c-mgzhb\" (UID: \"feba8fb8-7800-4069-9c0e-21be998399c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.343893 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.343863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cnv\" (UniqueName: \"kubernetes.io/projected/feba8fb8-7800-4069-9c0e-21be998399c2-kube-api-access-v9cnv\") pod \"openshift-lws-operator-fd8ccff4c-mgzhb\" (UID: \"feba8fb8-7800-4069-9c0e-21be998399c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.344032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.343928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feba8fb8-7800-4069-9c0e-21be998399c2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mgzhb\" (UID: \"feba8fb8-7800-4069-9c0e-21be998399c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.344282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.344263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feba8fb8-7800-4069-9c0e-21be998399c2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mgzhb\" (UID: \"feba8fb8-7800-4069-9c0e-21be998399c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.353046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.353020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cnv\" (UniqueName: \"kubernetes.io/projected/feba8fb8-7800-4069-9c0e-21be998399c2-kube-api-access-v9cnv\") pod \"openshift-lws-operator-fd8ccff4c-mgzhb\" (UID: \"feba8fb8-7800-4069-9c0e-21be998399c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.433239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.433210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"
Sep 14 10:28:26.457866 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.457832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6dgl2"]
Sep 14 10:28:26.461110 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:28:26.461087    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86fb51ca_701a_4cfb_aeee_16b8e1b3ee3e.slice/crio-e246b55dcc942c646386fec58f598c981fd436409741046954fcbc912c991169 WatchSource:0}: Error finding container e246b55dcc942c646386fec58f598c981fd436409741046954fcbc912c991169: Status 404 returned error can't find the container with id e246b55dcc942c646386fec58f598c981fd436409741046954fcbc912c991169
Sep 14 10:28:26.714404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.714333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb"]
Sep 14 10:28:26.717295 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:28:26.717272    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeba8fb8_7800_4069_9c0e_21be998399c2.slice/crio-ca19738f56d83142564e55ceb3eff48d190208ddbafd7a2753063b7a252bfdf4 WatchSource:0}: Error finding container ca19738f56d83142564e55ceb3eff48d190208ddbafd7a2753063b7a252bfdf4: Status 404 returned error can't find the container with id ca19738f56d83142564e55ceb3eff48d190208ddbafd7a2753063b7a252bfdf4
Sep 14 10:28:26.793707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.793675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb" event={"ID":"feba8fb8-7800-4069-9c0e-21be998399c2","Type":"ContainerStarted","Data":"ca19738f56d83142564e55ceb3eff48d190208ddbafd7a2753063b7a252bfdf4"}
Sep 14 10:28:26.794804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:26.794784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6dgl2" event={"ID":"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e","Type":"ContainerStarted","Data":"e246b55dcc942c646386fec58f598c981fd436409741046954fcbc912c991169"}
Sep 14 10:28:28.837852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:28.837817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:30.814759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:30.814719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb" event={"ID":"feba8fb8-7800-4069-9c0e-21be998399c2","Type":"ContainerStarted","Data":"3b14d2059b61f764224ead819b622e9b377f0460099d301262e50034bf224d94"}
Sep 14 10:28:30.816370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:30.816346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6dgl2" event={"ID":"86fb51ca-701a-4cfb-aeee-16b8e1b3ee3e","Type":"ContainerStarted","Data":"a0457ed82a8e769cebea6e89c6778f5ebaca94a03108533694fdfddd59c85d8a"}
Sep 14 10:28:30.833873 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:30.833812    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mgzhb" podStartSLOduration=0.867185295 podStartE2EDuration="4.833797457s" totalImagesPullingTime="3.966612162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:28:30.831871499 +0000 UTC m=+199.387059424" watchObservedRunningTime="2026-09-14 10:28:30.833797457 +0000 UTC m=+199.388985380"
Sep 14 10:28:31.560100 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:31.560067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:31.664316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:31.664280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:33.789165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:33.789121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:49.566309 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:49.566273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:49.720909 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:49.720874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:50.346367 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:50.346331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:51.144658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.144623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:51.340778 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.340744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:51.367006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.366958    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6dgl2" podStartSLOduration=22.192045188 podStartE2EDuration="26.366944324s" totalImagesPullingTime="4.174899136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:28:30.856049348 +0000 UTC m=+199.411237274" watchObservedRunningTime="2026-09-14 10:28:51.366944324 +0000 UTC m=+219.922132251"
Sep 14 10:28:51.367964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.367944    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"]
Sep 14 10:28:51.372021 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.372004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.376018 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.375991    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 14 10:28:51.376212 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.376192    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 14 10:28:51.376327 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.376312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 14 10:28:51.376433 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.376416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sfzpx\""
Sep 14 10:28:51.376519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.376504    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 14 10:28:51.394979 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.394925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"]
Sep 14 10:28:51.541483 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.541447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36389ec5-c492-4589-9823-53de660d9c77-webhook-cert\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.541667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.541493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdpv\" (UniqueName: \"kubernetes.io/projected/36389ec5-c492-4589-9823-53de660d9c77-kube-api-access-qrdpv\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.541667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.541536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36389ec5-c492-4589-9823-53de660d9c77-manifests\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.541667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.541645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36389ec5-c492-4589-9823-53de660d9c77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.541774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.541677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36389ec5-c492-4589-9823-53de660d9c77-tmp\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.583364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.583338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:51.642656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.642623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36389ec5-c492-4589-9823-53de660d9c77-tmp\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.642809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.642684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36389ec5-c492-4589-9823-53de660d9c77-webhook-cert\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.642809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.642721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdpv\" (UniqueName: \"kubernetes.io/projected/36389ec5-c492-4589-9823-53de660d9c77-kube-api-access-qrdpv\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.642917 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.642814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36389ec5-c492-4589-9823-53de660d9c77-manifests\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.642971 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.642918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36389ec5-c492-4589-9823-53de660d9c77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.643203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.643179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36389ec5-c492-4589-9823-53de660d9c77-manifests\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.645047 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.644992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36389ec5-c492-4589-9823-53de660d9c77-tmp\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.645137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.645121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36389ec5-c492-4589-9823-53de660d9c77-webhook-cert\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.645745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.645730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36389ec5-c492-4589-9823-53de660d9c77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.656051 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.656027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdpv\" (UniqueName: \"kubernetes.io/projected/36389ec5-c492-4589-9823-53de660d9c77-kube-api-access-qrdpv\") pod \"opendatahub-operator-controller-manager-757dcccd8b-fsnj2\" (UID: \"36389ec5-c492-4589-9823-53de660d9c77\") " pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.681972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.681943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:51.692532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.692505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:51.806149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.806114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:51.961404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.961327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:51.978603 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:51.978558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"]
Sep 14 10:28:51.982705 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:28:51.982682    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36389ec5_c492_4589_9823_53de660d9c77.slice/crio-2385a68ee5f1657ec800efb9dfa97764d900f3408d9adf3d4b03183d4f9a2f01 WatchSource:0}: Error finding container 2385a68ee5f1657ec800efb9dfa97764d900f3408d9adf3d4b03183d4f9a2f01: Status 404 returned error can't find the container with id 2385a68ee5f1657ec800efb9dfa97764d900f3408d9adf3d4b03183d4f9a2f01
Sep 14 10:28:52.058700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:52.058638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:52.162362 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:52.162326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:52.258763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:52.258696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:28:52.903905 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:52.903867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2" event={"ID":"36389ec5-c492-4589-9823-53de660d9c77","Type":"ContainerStarted","Data":"2385a68ee5f1657ec800efb9dfa97764d900f3408d9adf3d4b03183d4f9a2f01"}
Sep 14 10:28:54.914290 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:54.914258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2" event={"ID":"36389ec5-c492-4589-9823-53de660d9c77","Type":"ContainerStarted","Data":"461fb9658261f1f6a782ef40d6ff9c4e691db80031883a26ca6e1508e83b80e8"}
Sep 14 10:28:56.924144 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:56.924111    2563 generic.go:356] "Generic (PLEG): container finished" podID="36389ec5-c492-4589-9823-53de660d9c77" containerID="461fb9658261f1f6a782ef40d6ff9c4e691db80031883a26ca6e1508e83b80e8" exitCode=0
Sep 14 10:28:56.924499 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:56.924189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2" event={"ID":"36389ec5-c492-4589-9823-53de660d9c77","Type":"ContainerDied","Data":"461fb9658261f1f6a782ef40d6ff9c4e691db80031883a26ca6e1508e83b80e8"}
Sep 14 10:28:57.930086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:57.930049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2" event={"ID":"36389ec5-c492-4589-9823-53de660d9c77","Type":"ContainerStarted","Data":"b619b6ca8fd34545371c1705ff923fec77abafbb6975568e86e3f197edba2168"}
Sep 14 10:28:57.930482 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:57.930266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:28:57.964032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:28:57.963983    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2" podStartSLOduration=2.775307533 podStartE2EDuration="6.96396793s" totalImagesPullingTime="3.177962344s" totalInitContainerRuntime="1.010698053s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:28:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:28:57.962410527 +0000 UTC m=+226.517598451" watchObservedRunningTime="2026-09-14 10:28:57.96396793 +0000 UTC m=+226.519155856"
Sep 14 10:29:02.407592 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:02.407537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:02.592941 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:02.592910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:06.540291 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:06.540258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:06.730278 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:06.730248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:08.550027 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.549992    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"]
Sep 14 10:29:08.552866 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.552849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.555208 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.555189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 14 10:29:08.555973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.555947    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 14 10:29:08.556024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.555981    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mhq6q\""
Sep 14 10:29:08.556065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.555987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 14 10:29:08.565971 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.565948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"]
Sep 14 10:29:08.581801 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.581745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcf2aaad-ed53-43ff-8316-91b321fa72cf-cert\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.581896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.581816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fcf2aaad-ed53-43ff-8316-91b321fa72cf-manager-config\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.581896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.581851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7psz\" (UniqueName: \"kubernetes.io/projected/fcf2aaad-ed53-43ff-8316-91b321fa72cf-kube-api-access-s7psz\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.581896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.581886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcf2aaad-ed53-43ff-8316-91b321fa72cf-metrics-cert\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.683296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.683257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fcf2aaad-ed53-43ff-8316-91b321fa72cf-manager-config\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.683446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.683319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7psz\" (UniqueName: \"kubernetes.io/projected/fcf2aaad-ed53-43ff-8316-91b321fa72cf-kube-api-access-s7psz\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.683446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.683379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcf2aaad-ed53-43ff-8316-91b321fa72cf-metrics-cert\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.683446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.683423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcf2aaad-ed53-43ff-8316-91b321fa72cf-cert\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.684038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.684008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fcf2aaad-ed53-43ff-8316-91b321fa72cf-manager-config\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.686511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.686483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcf2aaad-ed53-43ff-8316-91b321fa72cf-cert\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.686611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.686593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcf2aaad-ed53-43ff-8316-91b321fa72cf-metrics-cert\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.695586 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.692013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7psz\" (UniqueName: \"kubernetes.io/projected/fcf2aaad-ed53-43ff-8316-91b321fa72cf-kube-api-access-s7psz\") pod \"lws-controller-manager-594c4c57b7-t7hw4\" (UID: \"fcf2aaad-ed53-43ff-8316-91b321fa72cf\") " pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.861610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.861502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:08.936258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:08.936219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-757dcccd8b-fsnj2"
Sep 14 10:29:09.148255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:09.148213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"]
Sep 14 10:29:09.151069 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:29:09.151041    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf2aaad_ed53_43ff_8316_91b321fa72cf.slice/crio-bd4d25dec3fdbf041c84a72a8f448ab1bea59efbc3e005375177acce460c4af4 WatchSource:0}: Error finding container bd4d25dec3fdbf041c84a72a8f448ab1bea59efbc3e005375177acce460c4af4: Status 404 returned error can't find the container with id bd4d25dec3fdbf041c84a72a8f448ab1bea59efbc3e005375177acce460c4af4
Sep 14 10:29:09.981076 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:09.981037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4" event={"ID":"fcf2aaad-ed53-43ff-8316-91b321fa72cf","Type":"ContainerStarted","Data":"bd4d25dec3fdbf041c84a72a8f448ab1bea59efbc3e005375177acce460c4af4"}
Sep 14 10:29:10.986404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:10.986368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4" event={"ID":"fcf2aaad-ed53-43ff-8316-91b321fa72cf","Type":"ContainerStarted","Data":"0fa4e401826f1acf2cef5925936679c1e71a5b9da47643b8fceb763f1b5eb09d"}
Sep 14 10:29:10.986760 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:10.986458    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:11.016357 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:11.016312    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4" podStartSLOduration=1.803293358 podStartE2EDuration="3.016297862s" totalImagesPullingTime="1.213004504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:29:11.014885062 +0000 UTC m=+239.570072988" watchObservedRunningTime="2026-09-14 10:29:11.016297862 +0000 UTC m=+239.571485787"
Sep 14 10:29:11.275111 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:11.275021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:11.479124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:11.479085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:21.992697 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:21.992661    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-594c4c57b7-t7hw4"
Sep 14 10:29:22.160241 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.160212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:22.240471 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.240438    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"]
Sep 14 10:29:22.244147 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.244081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.247305 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.247281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 14 10:29:22.248279 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.248252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 14 10:29:22.252257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.252237    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-snh5n\""
Sep 14 10:29:22.252409 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.252393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 14 10:29:22.252478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.252403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 14 10:29:22.285547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.285514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"]
Sep 14 10:29:22.285834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.285812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5246m\" (UniqueName: \"kubernetes.io/projected/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-kube-api-access-5246m\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.285879 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.285854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tls-certs\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.285879 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.285869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tmp\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.386754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.386725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tls-certs\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.386754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.386755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tmp\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.387010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.386841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5246m\" (UniqueName: \"kubernetes.io/projected/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-kube-api-access-5246m\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.387010 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:29:22.386883    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 14 10:29:22.387010 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:29:22.386947    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tls-certs podName:b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33 nodeName:}" failed. No retries permitted until 2026-09-14 10:29:22.886932097 +0000 UTC m=+251.442120001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tls-certs") pod "kube-auth-proxy-cb779b95-ww4qf" (UID: "b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33") : secret "kube-auth-proxy-tls" not found
Sep 14 10:29:22.389147 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.389125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tmp\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.404709 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.404689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5246m\" (UniqueName: \"kubernetes.io/projected/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-kube-api-access-5246m\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.511886 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.511802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:22.659983 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.659953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:22.792888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.792803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:22.890679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.890646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tls-certs\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:22.890840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.890681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:22.893290 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:22.893261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33-tls-certs\") pod \"kube-auth-proxy-cb779b95-ww4qf\" (UID: \"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33\") " pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:23.084713 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:23.084627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:23.154079 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:23.154045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"
Sep 14 10:29:23.438483 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:23.438448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb779b95-ww4qf"]
Sep 14 10:29:23.441682 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:29:23.441654    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47b4f3b_e44c_4efa_bbc2_ec5ae7520c33.slice/crio-0d485ba9cec3b733c9128c286cc023f1fef2e34c6c2e7e80b824fcc1963b2053 WatchSource:0}: Error finding container 0d485ba9cec3b733c9128c286cc023f1fef2e34c6c2e7e80b824fcc1963b2053: Status 404 returned error can't find the container with id 0d485ba9cec3b733c9128c286cc023f1fef2e34c6c2e7e80b824fcc1963b2053
Sep 14 10:29:23.441682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:23.441662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:24.043379 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:24.043333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf" event={"ID":"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33","Type":"ContainerStarted","Data":"0d485ba9cec3b733c9128c286cc023f1fef2e34c6c2e7e80b824fcc1963b2053"}
Sep 14 10:29:24.045275 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:24.045183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:24.205532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:24.205397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:25.416227 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:25.416193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:27.060544 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:27.060508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf" event={"ID":"b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33","Type":"ContainerStarted","Data":"5a1e19fcc92a0f03317612519c1cccdb43880495fac19fa8c6fb310c6ee413e0"}
Sep 14 10:29:27.079168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:27.079114    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cb779b95-ww4qf" podStartSLOduration=2.372435994 podStartE2EDuration="5.079093442s" totalImagesPullingTime="2.706657448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:29:27.07775422 +0000 UTC m=+255.632942146" watchObservedRunningTime="2026-09-14 10:29:27.079093442 +0000 UTC m=+255.634281369"
Sep 14 10:29:27.275088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:27.275018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:27.349245 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:27.349115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:28.984823 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:28.984791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:30.752865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:30.752798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:30.989860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:30.989823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:32.694493 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:32.694463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:33.072749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:33.072659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:33.414374 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:33.414347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:34.853401 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:34.853369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:35.007929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:35.007894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:35.161939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:35.161911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:35.326961 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:35.326924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:36.310441 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:36.310407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:36.476703 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:36.476666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:29:36.643898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:29:36.643848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:11.802913 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:11.802884    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 14 10:30:16.091335 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:16.091301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:17.724744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:17.724711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:23.393874 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:23.393838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:23.491126 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:23.491091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:39.334917 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:39.334885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:39.401135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:39.401108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:51.933803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:51.933774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:52.944775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:52.944747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:53.983994 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:53.983966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:57.446502 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:57.446472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:57.553510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:57.553474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:58.764912 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:58.764877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:30:59.361661 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:30:59.361606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:00.907816 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:00.907783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:01.140965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.140926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:01.231850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.231776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:01.640386 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.640357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:01.742967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.742940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:01.894977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.894905    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p9vjv"]
Sep 14 10:31:01.898061 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.898043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv"
Sep 14 10:31:01.900306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.900287    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 14 10:31:01.900417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.900290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 14 10:31:01.900701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.900685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-c7cf4\""
Sep 14 10:31:01.906674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:01.906653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p9vjv"]
Sep 14 10:31:02.068781 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:02.068753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnns\" (UniqueName: \"kubernetes.io/projected/ad840875-a2e3-4c37-846b-6cae5dd1d7c1-kube-api-access-fvnns\") pod \"authorino-operator-76b478bd78-p9vjv\" (UID: \"ad840875-a2e3-4c37-846b-6cae5dd1d7c1\") " pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv"
Sep 14 10:31:02.169762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:02.169705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnns\" (UniqueName: \"kubernetes.io/projected/ad840875-a2e3-4c37-846b-6cae5dd1d7c1-kube-api-access-fvnns\") pod \"authorino-operator-76b478bd78-p9vjv\" (UID: \"ad840875-a2e3-4c37-846b-6cae5dd1d7c1\") " pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv"
Sep 14 10:31:02.178002 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:02.177978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnns\" (UniqueName: \"kubernetes.io/projected/ad840875-a2e3-4c37-846b-6cae5dd1d7c1-kube-api-access-fvnns\") pod \"authorino-operator-76b478bd78-p9vjv\" (UID: \"ad840875-a2e3-4c37-846b-6cae5dd1d7c1\") " pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv"
Sep 14 10:31:02.208020 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:02.207987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv"
Sep 14 10:31:02.482476 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:02.482404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p9vjv"]
Sep 14 10:31:02.485214 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:31:02.485178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad840875_a2e3_4c37_846b_6cae5dd1d7c1.slice/crio-ef44efda12268712ccdcf485a638e325a6f8a85ad57ba4f19f0db6323f3cf449 WatchSource:0}: Error finding container ef44efda12268712ccdcf485a638e325a6f8a85ad57ba4f19f0db6323f3cf449: Status 404 returned error can't find the container with id ef44efda12268712ccdcf485a638e325a6f8a85ad57ba4f19f0db6323f3cf449
Sep 14 10:31:02.487453 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:02.487436    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:31:02.990222 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:02.990189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:03.468731 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:03.468703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv" event={"ID":"ad840875-a2e3-4c37-846b-6cae5dd1d7c1","Type":"ContainerStarted","Data":"ef44efda12268712ccdcf485a638e325a6f8a85ad57ba4f19f0db6323f3cf449"}
Sep 14 10:31:05.479747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:05.479719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv" event={"ID":"ad840875-a2e3-4c37-846b-6cae5dd1d7c1","Type":"ContainerStarted","Data":"db12c561da2c2131c9c8bbe05e79e30a21f3ce0acc0fc59c8bc67533cee539fe"}
Sep 14 10:31:05.480009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:05.479776    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv"
Sep 14 10:31:05.499478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:05.499430    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv" podStartSLOduration=1.610855104 podStartE2EDuration="4.499415569s" totalImagesPullingTime="2.888560465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:31:05.498317384 +0000 UTC m=+354.053505309" watchObservedRunningTime="2026-09-14 10:31:05.499415569 +0000 UTC m=+354.054603495"
Sep 14 10:31:05.990435 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:05.990389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:06.145641 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:06.145607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:06.361476 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:06.361392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:11.956277 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:11.956239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:12.056893 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:12.056858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:12.157139 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:12.157109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:12.257620 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:12.257530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:12.540305 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:12.540235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:12.599584 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:12.599541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:13.137623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:13.137589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:13.246920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:13.246891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:16.487355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:16.487321    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-p9vjv"
Sep 14 10:31:17.800056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:17.800018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:17.914609 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:17.914553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:18.790687 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:18.790654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:19.642312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:19.642278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:21.370995 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:21.370960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:21.471517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:21.471488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:21.570746 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:21.570717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:21.671861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:21.671833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:25.179353 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:25.179314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:25.281503 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:25.281464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:29.183282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:29.183245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:29.286021 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:29.285992    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:34.342491 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:34.342453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:34.440446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:34.440414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:34.542557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:34.542517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:43.164313 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:43.164254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:43.336584 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:43.336531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:45.602364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:45.602321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:45.703156 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:45.703116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:46.468896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:46.468866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:46.708952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:46.708922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:50.425910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:50.425880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:50.526218 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:50.526184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:55.571255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:55.571223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:55.672101 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:55.672075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:55.975768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:55.975737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:56.044031 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:56.044003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:56.271692 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:56.271628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:31:56.382037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:31:56.382011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:01.978151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:01.978121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:02.113781 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.113752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:02.229865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.229791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:02.521222 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.521146    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:32:02.527829 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.527806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.530961 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.530944    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 14 10:32:02.531054 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.530985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tk8d8\""
Sep 14 10:32:02.536359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.536339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:32:02.575656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.575634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:32:02.652516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.652496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzl2\" (UniqueName: \"kubernetes.io/projected/3a9f2178-5ca9-4d4f-a1a0-7783d4b493da-kube-api-access-rmzl2\") pod \"limitador-limitador-84994f7ffc-lldrw\" (UID: \"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.652614 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.652544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a9f2178-5ca9-4d4f-a1a0-7783d4b493da-config-file\") pod \"limitador-limitador-84994f7ffc-lldrw\" (UID: \"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.753125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.753098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzl2\" (UniqueName: \"kubernetes.io/projected/3a9f2178-5ca9-4d4f-a1a0-7783d4b493da-kube-api-access-rmzl2\") pod \"limitador-limitador-84994f7ffc-lldrw\" (UID: \"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.753244 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.753158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a9f2178-5ca9-4d4f-a1a0-7783d4b493da-config-file\") pod \"limitador-limitador-84994f7ffc-lldrw\" (UID: \"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.753699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.753682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a9f2178-5ca9-4d4f-a1a0-7783d4b493da-config-file\") pod \"limitador-limitador-84994f7ffc-lldrw\" (UID: \"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.765700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.765680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzl2\" (UniqueName: \"kubernetes.io/projected/3a9f2178-5ca9-4d4f-a1a0-7783d4b493da-kube-api-access-rmzl2\") pod \"limitador-limitador-84994f7ffc-lldrw\" (UID: \"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.839210 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.839155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:02.908740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:02.908705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:03.001606 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.001562    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsnvr"]
Sep 14 10:32:03.002191 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.002170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:03.005268 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.005251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsnvr"
Sep 14 10:32:03.007252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.007226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2sn7g\""
Sep 14 10:32:03.017804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.017783    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsnvr"]
Sep 14 10:32:03.114240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.114177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:32:03.117373 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:32:03.117350    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9f2178_5ca9_4d4f_a1a0_7783d4b493da.slice/crio-e68f42b273eab3393b82298bbb578ca3f0c3e3519eda33b831226c2e6b280225 WatchSource:0}: Error finding container e68f42b273eab3393b82298bbb578ca3f0c3e3519eda33b831226c2e6b280225: Status 404 returned error can't find the container with id e68f42b273eab3393b82298bbb578ca3f0c3e3519eda33b831226c2e6b280225
Sep 14 10:32:03.157551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.157526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8lb\" (UniqueName: \"kubernetes.io/projected/5a7501f7-d7f2-4286-92fc-f28cf002377e-kube-api-access-bm8lb\") pod \"authorino-59c974bd48-wsnvr\" (UID: \"5a7501f7-d7f2-4286-92fc-f28cf002377e\") " pod="kuadrant-system/authorino-59c974bd48-wsnvr"
Sep 14 10:32:03.258710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.258671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8lb\" (UniqueName: \"kubernetes.io/projected/5a7501f7-d7f2-4286-92fc-f28cf002377e-kube-api-access-bm8lb\") pod \"authorino-59c974bd48-wsnvr\" (UID: \"5a7501f7-d7f2-4286-92fc-f28cf002377e\") " pod="kuadrant-system/authorino-59c974bd48-wsnvr"
Sep 14 10:32:03.268583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.268551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8lb\" (UniqueName: \"kubernetes.io/projected/5a7501f7-d7f2-4286-92fc-f28cf002377e-kube-api-access-bm8lb\") pod \"authorino-59c974bd48-wsnvr\" (UID: \"5a7501f7-d7f2-4286-92fc-f28cf002377e\") " pod="kuadrant-system/authorino-59c974bd48-wsnvr"
Sep 14 10:32:03.314170 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.314145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsnvr"
Sep 14 10:32:03.581788 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.581762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsnvr"]
Sep 14 10:32:03.584211 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:32:03.584186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a7501f7_d7f2_4286_92fc_f28cf002377e.slice/crio-db532585887e03c7a53e2f143e580ee7e333b6d92e5892582603cbfc694b36af WatchSource:0}: Error finding container db532585887e03c7a53e2f143e580ee7e333b6d92e5892582603cbfc694b36af: Status 404 returned error can't find the container with id db532585887e03c7a53e2f143e580ee7e333b6d92e5892582603cbfc694b36af
Sep 14 10:32:03.730490 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.730459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsnvr" event={"ID":"5a7501f7-d7f2-4286-92fc-f28cf002377e","Type":"ContainerStarted","Data":"db532585887e03c7a53e2f143e580ee7e333b6d92e5892582603cbfc694b36af"}
Sep 14 10:32:03.731600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:03.731561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw" event={"ID":"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da","Type":"ContainerStarted","Data":"e68f42b273eab3393b82298bbb578ca3f0c3e3519eda33b831226c2e6b280225"}
Sep 14 10:32:04.586901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:04.586862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:04.685421 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:04.685396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:04.737964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:04.737938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw" event={"ID":"3a9f2178-5ca9-4d4f-a1a0-7783d4b493da","Type":"ContainerStarted","Data":"e4232a418b2bf65f32ace760bfd5a66fa09e16194eb8f6d0cabcb20fdbcea264"}
Sep 14 10:32:04.738117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:04.738101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:04.755938 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:04.755899    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw" podStartSLOduration=1.225222081 podStartE2EDuration="2.755883315s" totalImagesPullingTime="1.530661234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:32:04.753829134 +0000 UTC m=+413.309017059" watchObservedRunningTime="2026-09-14 10:32:04.755883315 +0000 UTC m=+413.311071240"
Sep 14 10:32:05.744211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:05.744169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsnvr" event={"ID":"5a7501f7-d7f2-4286-92fc-f28cf002377e","Type":"ContainerStarted","Data":"d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac"}
Sep 14 10:32:05.766678 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:05.766634    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-wsnvr" podStartSLOduration=2.164357061 podStartE2EDuration="3.766619034s" totalImagesPullingTime="1.602261973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:32:05.75964962 +0000 UTC m=+414.314837541" watchObservedRunningTime="2026-09-14 10:32:05.766619034 +0000 UTC m=+414.321806958"
Sep 14 10:32:07.022202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:07.022172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:07.124584 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:07.124529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:11.787138 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:11.787106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:11.899410 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:11.899374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:15.745075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:15.745047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-lldrw"
Sep 14 10:32:20.655756 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:20.655704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:20.750875 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:20.750829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:21.188714 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:21.188684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:21.454906 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:21.454834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:21.556012 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:21.555973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:31.589346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.589308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:31.800228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.800192    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-knbqt"]
Sep 14 10:32:31.805965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.805939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-knbqt"
Sep 14 10:32:31.812973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.812643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-knbqt"]
Sep 14 10:32:31.836210 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.836188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:31.886585 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.884353    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-knbqt"]
Sep 14 10:32:31.886585 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:32:31.884952    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2rnn9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-knbqt" podUID="69226b5f-f9ee-4023-b273-5e5deea8333f"
Sep 14 10:32:31.890648 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.890625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnn9\" (UniqueName: \"kubernetes.io/projected/69226b5f-f9ee-4023-b273-5e5deea8333f-kube-api-access-2rnn9\") pod \"authorino-557f876f69-knbqt\" (UID: \"69226b5f-f9ee-4023-b273-5e5deea8333f\") " pod="kuadrant-system/authorino-557f876f69-knbqt"
Sep 14 10:32:31.919404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.919380    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wcrc"]
Sep 14 10:32:31.923361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.923345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:31.925266 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.925248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 14 10:32:31.928052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.928031    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wcrc"]
Sep 14 10:32:31.991596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.991545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs68x\" (UniqueName: \"kubernetes.io/projected/9baa2fc0-fb5d-49ae-b353-ce7161960f11-kube-api-access-fs68x\") pod \"authorino-6d4dfc649f-4wcrc\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") " pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:31.991729 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.991652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9baa2fc0-fb5d-49ae-b353-ce7161960f11-tls-cert\") pod \"authorino-6d4dfc649f-4wcrc\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") " pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:31.991789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:31.991767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnn9\" (UniqueName: \"kubernetes.io/projected/69226b5f-f9ee-4023-b273-5e5deea8333f-kube-api-access-2rnn9\") pod \"authorino-557f876f69-knbqt\" (UID: \"69226b5f-f9ee-4023-b273-5e5deea8333f\") " pod="kuadrant-system/authorino-557f876f69-knbqt"
Sep 14 10:32:32.002193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.002171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnn9\" (UniqueName: \"kubernetes.io/projected/69226b5f-f9ee-4023-b273-5e5deea8333f-kube-api-access-2rnn9\") pod \"authorino-557f876f69-knbqt\" (UID: \"69226b5f-f9ee-4023-b273-5e5deea8333f\") " pod="kuadrant-system/authorino-557f876f69-knbqt"
Sep 14 10:32:32.064800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.064774    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wcrc"]
Sep 14 10:32:32.065025 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:32:32.065005    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fs68x tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-4wcrc" podUID="9baa2fc0-fb5d-49ae-b353-ce7161960f11"
Sep 14 10:32:32.092900 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.092876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs68x\" (UniqueName: \"kubernetes.io/projected/9baa2fc0-fb5d-49ae-b353-ce7161960f11-kube-api-access-fs68x\") pod \"authorino-6d4dfc649f-4wcrc\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") " pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:32.093009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.092909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9baa2fc0-fb5d-49ae-b353-ce7161960f11-tls-cert\") pod \"authorino-6d4dfc649f-4wcrc\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") " pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:32.093506 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.093483    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d94c75687-jpfpj"]
Sep 14 10:32:32.095392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.095373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9baa2fc0-fb5d-49ae-b353-ce7161960f11-tls-cert\") pod \"authorino-6d4dfc649f-4wcrc\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") " pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:32.097645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.097627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.101964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.101936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs68x\" (UniqueName: \"kubernetes.io/projected/9baa2fc0-fb5d-49ae-b353-ce7161960f11-kube-api-access-fs68x\") pod \"authorino-6d4dfc649f-4wcrc\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") " pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:32.104336 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.104318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d94c75687-jpfpj"]
Sep 14 10:32:32.194243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.194191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgjn\" (UniqueName: \"kubernetes.io/projected/e96551db-f075-49b9-8b6f-1e205b60740e-kube-api-access-4bgjn\") pod \"authorino-6d94c75687-jpfpj\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") " pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.194243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.194225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e96551db-f075-49b9-8b6f-1e205b60740e-tls-cert\") pod \"authorino-6d94c75687-jpfpj\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") " pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.295326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.295304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgjn\" (UniqueName: \"kubernetes.io/projected/e96551db-f075-49b9-8b6f-1e205b60740e-kube-api-access-4bgjn\") pod \"authorino-6d94c75687-jpfpj\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") " pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.295421 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.295341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e96551db-f075-49b9-8b6f-1e205b60740e-tls-cert\") pod \"authorino-6d94c75687-jpfpj\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") " pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.297605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.297590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e96551db-f075-49b9-8b6f-1e205b60740e-tls-cert\") pod \"authorino-6d94c75687-jpfpj\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") " pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.305458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.305440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgjn\" (UniqueName: \"kubernetes.io/projected/e96551db-f075-49b9-8b6f-1e205b60740e-kube-api-access-4bgjn\") pod \"authorino-6d94c75687-jpfpj\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") " pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.420448 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.420424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:32:32.692992 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.692968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d94c75687-jpfpj"]
Sep 14 10:32:32.695122 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:32:32.695097    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode96551db_f075_49b9_8b6f_1e205b60740e.slice/crio-aca473743d8c378ce99d506e7afe5bd5a18d480ae19d6c810b9d5fad6a7bdac9 WatchSource:0}: Error finding container aca473743d8c378ce99d506e7afe5bd5a18d480ae19d6c810b9d5fad6a7bdac9: Status 404 returned error can't find the container with id aca473743d8c378ce99d506e7afe5bd5a18d480ae19d6c810b9d5fad6a7bdac9
Sep 14 10:32:32.861385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.861314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d94c75687-jpfpj" event={"ID":"e96551db-f075-49b9-8b6f-1e205b60740e","Type":"ContainerStarted","Data":"aca473743d8c378ce99d506e7afe5bd5a18d480ae19d6c810b9d5fad6a7bdac9"}
Sep 14 10:32:32.861385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.861340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-knbqt"
Sep 14 10:32:32.861385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.861332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:32.869870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.869854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-knbqt"
Sep 14 10:32:32.872324 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.872309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:32.899588 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.899545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9baa2fc0-fb5d-49ae-b353-ce7161960f11-tls-cert\") pod \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") "
Sep 14 10:32:32.899680 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.899601    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rnn9\" (UniqueName: \"kubernetes.io/projected/69226b5f-f9ee-4023-b273-5e5deea8333f-kube-api-access-2rnn9\") pod \"69226b5f-f9ee-4023-b273-5e5deea8333f\" (UID: \"69226b5f-f9ee-4023-b273-5e5deea8333f\") "
Sep 14 10:32:32.899680 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.899665    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs68x\" (UniqueName: \"kubernetes.io/projected/9baa2fc0-fb5d-49ae-b353-ce7161960f11-kube-api-access-fs68x\") pod \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\" (UID: \"9baa2fc0-fb5d-49ae-b353-ce7161960f11\") "
Sep 14 10:32:32.901662 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.901637    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69226b5f-f9ee-4023-b273-5e5deea8333f-kube-api-access-2rnn9" (OuterVolumeSpecName: "kube-api-access-2rnn9") pod "69226b5f-f9ee-4023-b273-5e5deea8333f" (UID: "69226b5f-f9ee-4023-b273-5e5deea8333f"). InnerVolumeSpecName "kube-api-access-2rnn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:32:32.901740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.901706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9baa2fc0-fb5d-49ae-b353-ce7161960f11-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9baa2fc0-fb5d-49ae-b353-ce7161960f11" (UID: "9baa2fc0-fb5d-49ae-b353-ce7161960f11"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:32:32.901740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:32.901707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9baa2fc0-fb5d-49ae-b353-ce7161960f11-kube-api-access-fs68x" (OuterVolumeSpecName: "kube-api-access-fs68x") pod "9baa2fc0-fb5d-49ae-b353-ce7161960f11" (UID: "9baa2fc0-fb5d-49ae-b353-ce7161960f11"). InnerVolumeSpecName "kube-api-access-fs68x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:32:33.000922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.000894    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rnn9\" (UniqueName: \"kubernetes.io/projected/69226b5f-f9ee-4023-b273-5e5deea8333f-kube-api-access-2rnn9\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:32:33.000922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.000917    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs68x\" (UniqueName: \"kubernetes.io/projected/9baa2fc0-fb5d-49ae-b353-ce7161960f11-kube-api-access-fs68x\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:32:33.000922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.000926    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9baa2fc0-fb5d-49ae-b353-ce7161960f11-tls-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:32:33.867097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.867066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-4wcrc"
Sep 14 10:32:33.867538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.867104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d94c75687-jpfpj" event={"ID":"e96551db-f075-49b9-8b6f-1e205b60740e","Type":"ContainerStarted","Data":"b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83"}
Sep 14 10:32:33.867538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.867229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-knbqt"
Sep 14 10:32:33.883854 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.883813    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d94c75687-jpfpj" podStartSLOduration=1.223932566 podStartE2EDuration="1.883800126s" totalImagesPullingTime="659.86756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:32:33.88249671 +0000 UTC m=+442.437684640" watchObservedRunningTime="2026-09-14 10:32:33.883800126 +0000 UTC m=+442.438988052"
Sep 14 10:32:33.911141 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.911111    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-knbqt"]
Sep 14 10:32:33.919179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.919156    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-knbqt"]
Sep 14 10:32:33.919308 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.919190    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsnvr"]
Sep 14 10:32:33.919363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.919322    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-wsnvr" podUID="5a7501f7-d7f2-4286-92fc-f28cf002377e" containerName="authorino" containerID="cri-o://d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac" gracePeriod=30
Sep 14 10:32:33.945897 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.945867    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wcrc"]
Sep 14 10:32:33.951623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:33.951594    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-4wcrc"]
Sep 14 10:32:34.154932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.154913    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsnvr"
Sep 14 10:32:34.210696 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.210665    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm8lb\" (UniqueName: \"kubernetes.io/projected/5a7501f7-d7f2-4286-92fc-f28cf002377e-kube-api-access-bm8lb\") pod \"5a7501f7-d7f2-4286-92fc-f28cf002377e\" (UID: \"5a7501f7-d7f2-4286-92fc-f28cf002377e\") "
Sep 14 10:32:34.212795 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.212772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7501f7-d7f2-4286-92fc-f28cf002377e-kube-api-access-bm8lb" (OuterVolumeSpecName: "kube-api-access-bm8lb") pod "5a7501f7-d7f2-4286-92fc-f28cf002377e" (UID: "5a7501f7-d7f2-4286-92fc-f28cf002377e"). InnerVolumeSpecName "kube-api-access-bm8lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:32:34.312138 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.312114    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm8lb\" (UniqueName: \"kubernetes.io/projected/5a7501f7-d7f2-4286-92fc-f28cf002377e-kube-api-access-bm8lb\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:32:34.872286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.872252    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a7501f7-d7f2-4286-92fc-f28cf002377e" containerID="d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac" exitCode=0
Sep 14 10:32:34.872745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.872312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsnvr"
Sep 14 10:32:34.872745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.872333    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsnvr" event={"ID":"5a7501f7-d7f2-4286-92fc-f28cf002377e","Type":"ContainerDied","Data":"d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac"}
Sep 14 10:32:34.872745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.872375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsnvr" event={"ID":"5a7501f7-d7f2-4286-92fc-f28cf002377e","Type":"ContainerDied","Data":"db532585887e03c7a53e2f143e580ee7e333b6d92e5892582603cbfc694b36af"}
Sep 14 10:32:34.872745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.872391    2563 scope.go:117] "RemoveContainer" containerID="d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac"
Sep 14 10:32:34.880700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.880684    2563 scope.go:117] "RemoveContainer" containerID="d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac"
Sep 14 10:32:34.880955 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:32:34.880937    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac\": container with ID starting with d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac not found: ID does not exist" containerID="d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac"
Sep 14 10:32:34.880991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.880963    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac"} err="failed to get container status \"d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac\": rpc error: code = NotFound desc = could not find container \"d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac\": container with ID starting with d7cd003ac3085a375c42ccd88a0ee317da01280e15d0f207f216e934ae5c73ac not found: ID does not exist"
Sep 14 10:32:34.891301 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.891273    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsnvr"]
Sep 14 10:32:34.895855 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:34.895835    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsnvr"]
Sep 14 10:32:35.917270 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:35.917241    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a7501f7-d7f2-4286-92fc-f28cf002377e" path="/var/lib/kubelet/pods/5a7501f7-d7f2-4286-92fc-f28cf002377e/volumes"
Sep 14 10:32:35.917659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:35.917551    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69226b5f-f9ee-4023-b273-5e5deea8333f" path="/var/lib/kubelet/pods/69226b5f-f9ee-4023-b273-5e5deea8333f/volumes"
Sep 14 10:32:35.917789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:35.917776    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9baa2fc0-fb5d-49ae-b353-ce7161960f11" path="/var/lib/kubelet/pods/9baa2fc0-fb5d-49ae-b353-ce7161960f11/volumes"
Sep 14 10:32:38.775334 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:38.775303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:38.868335 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:38.868306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:38.961069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:38.961037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:39.040459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:39.040388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:39.203733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:39.203701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:42.763438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:42.763404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:32:42.885710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:32:42.885677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:14.213980 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:14.213925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:14.362929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:14.362895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:15.677794 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:15.677759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:15.804582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:15.804539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:15.904753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:15.904710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:16.295296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:16.295264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:16.400300 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:16.400257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:16.499645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:16.499607    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:33:16.511914 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:16.511879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:24.920768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:24.920730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:27.227415 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:27.227381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:27.353698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:27.353669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:33.520080 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:33.520036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:34.069513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:34.069473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:33:44.257631 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:44.257594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:44.980921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:44.980887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:33:54.709483 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:54.709448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:33:55.873954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:33:55.873912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:34:00.988809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:00.988772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:34:02.266583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:02.266531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:34:14.733800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:14.733699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:34:16.292059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:16.292017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:34:20.807496 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:20.807459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:34:27.064209 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.064177    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"]
Sep 14 10:34:27.064646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.064514    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7501f7-d7f2-4286-92fc-f28cf002377e" containerName="authorino"
Sep 14 10:34:27.064646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.064525    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7501f7-d7f2-4286-92fc-f28cf002377e" containerName="authorino"
Sep 14 10:34:27.064646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.064625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a7501f7-d7f2-4286-92fc-f28cf002377e" containerName="authorino"
Sep 14 10:34:27.067616 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.067597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.069793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.069771    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 14 10:34:27.069878 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.069801    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 14 10:34:27.070368 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.070351    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 14 10:34:27.070682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.070666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f5d8v\""
Sep 14 10:34:27.078801 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.078778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"]
Sep 14 10:34:27.091086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.091064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:34:27.128140 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.128103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2xm\" (UniqueName: \"kubernetes.io/projected/5dc770b8-ce2f-4926-b03b-c914e0672485-kube-api-access-fb2xm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.128285 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.128186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.128285 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.128241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dc770b8-ce2f-4926-b03b-c914e0672485-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.128285 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.128281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.128438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.128342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.128438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.128380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.229877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.229839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.229877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.229879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.230112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.229929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2xm\" (UniqueName: \"kubernetes.io/projected/5dc770b8-ce2f-4926-b03b-c914e0672485-kube-api-access-fb2xm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.230112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.229963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.230112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.230001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dc770b8-ce2f-4926-b03b-c914e0672485-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.230112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.230048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.230345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.230322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.230387 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.230358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.230421 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.230393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.232360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.232332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dc770b8-ce2f-4926-b03b-c914e0672485-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.232612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.232595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dc770b8-ce2f-4926-b03b-c914e0672485-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.237343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.237321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2xm\" (UniqueName: \"kubernetes.io/projected/5dc770b8-ce2f-4926-b03b-c914e0672485-kube-api-access-fb2xm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wqjk2\" (UID: \"5dc770b8-ce2f-4926-b03b-c914e0672485\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.379295 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.379274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:27.650170 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:27.650104    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"]
Sep 14 10:34:27.652982 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:34:27.652941    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc770b8_ce2f_4926_b03b_c914e0672485.slice/crio-4ca0644330be9922b7abb8ba9780903c15a0a997cd3f132a1e04bfe653b16574 WatchSource:0}: Error finding container 4ca0644330be9922b7abb8ba9780903c15a0a997cd3f132a1e04bfe653b16574: Status 404 returned error can't find the container with id 4ca0644330be9922b7abb8ba9780903c15a0a997cd3f132a1e04bfe653b16574
Sep 14 10:34:28.369148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:28.369106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2" event={"ID":"5dc770b8-ce2f-4926-b03b-c914e0672485","Type":"ContainerStarted","Data":"4ca0644330be9922b7abb8ba9780903c15a0a997cd3f132a1e04bfe653b16574"}
Sep 14 10:34:28.963640 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:28.963599    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:34:29.066158 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.066127    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"]
Sep 14 10:34:29.069781 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.069755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.071782 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.071757    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 14 10:34:29.079774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.079747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"]
Sep 14 10:34:29.150234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.150200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnzr\" (UniqueName: \"kubernetes.io/projected/f13b256c-7be8-4d28-8fbd-da2b69adeff6-kube-api-access-7jnzr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.150387 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.150264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13b256c-7be8-4d28-8fbd-da2b69adeff6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.150387 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.150345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.150460 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.150389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.150493 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.150458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.150526 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.150505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.214357 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.214265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:34:29.251393 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.251393 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.251665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnzr\" (UniqueName: \"kubernetes.io/projected/f13b256c-7be8-4d28-8fbd-da2b69adeff6-kube-api-access-7jnzr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.251665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13b256c-7be8-4d28-8fbd-da2b69adeff6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.251665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.251812 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.251865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.251959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.251940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.252034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.252011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.253970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.253949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f13b256c-7be8-4d28-8fbd-da2b69adeff6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.254084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.254069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f13b256c-7be8-4d28-8fbd-da2b69adeff6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.262493 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.262464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnzr\" (UniqueName: \"kubernetes.io/projected/f13b256c-7be8-4d28-8fbd-da2b69adeff6-kube-api-access-7jnzr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4\" (UID: \"f13b256c-7be8-4d28-8fbd-da2b69adeff6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.375455 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.375420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2" event={"ID":"5dc770b8-ce2f-4926-b03b-c914e0672485","Type":"ContainerStarted","Data":"ab6da98be4f0f78e8285fd9f151dd6a93c52d51a2c5dfc939e27e467ddcc26ea"}
Sep 14 10:34:29.380649 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.380630    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:29.396959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.396897    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2" podStartSLOduration=1.4277510310000001 podStartE2EDuration="2.39687627s" totalImagesPullingTime="969.125239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:34:29.396359094 +0000 UTC m=+557.951547021" watchObservedRunningTime="2026-09-14 10:34:29.39687627 +0000 UTC m=+557.952064197"
Sep 14 10:34:29.657824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:29.657792    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"]
Sep 14 10:34:29.661146 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:34:29.661117    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf13b256c_7be8_4d28_8fbd_da2b69adeff6.slice/crio-6681e24dead855ff2531acd073252f0001db58fb957b857c37601aad68a5ede1 WatchSource:0}: Error finding container 6681e24dead855ff2531acd073252f0001db58fb957b857c37601aad68a5ede1: Status 404 returned error can't find the container with id 6681e24dead855ff2531acd073252f0001db58fb957b857c37601aad68a5ede1
Sep 14 10:34:30.381533 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:30.381493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4" event={"ID":"f13b256c-7be8-4d28-8fbd-da2b69adeff6","Type":"ContainerStarted","Data":"b63269971f41fd5df227ce2e586bf5293b9228cc39da8cea06aad3e9ff11e05a"}
Sep 14 10:34:30.381533 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:30.381536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4" event={"ID":"f13b256c-7be8-4d28-8fbd-da2b69adeff6","Type":"ContainerStarted","Data":"6681e24dead855ff2531acd073252f0001db58fb957b857c37601aad68a5ede1"}
Sep 14 10:34:30.402133 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:30.402088    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4" podStartSLOduration=1.23244169 podStartE2EDuration="1.402073013s" totalImagesPullingTime="169.631323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:34:30.400691721 +0000 UTC m=+558.955879659" watchObservedRunningTime="2026-09-14 10:34:30.402073013 +0000 UTC m=+558.957260938"
Sep 14 10:34:36.369001 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:36.368970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:34:37.380397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:37.380360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:37.380835 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:37.380450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:37.392124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:37.392105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:37.424065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:37.424035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wqjk2"
Sep 14 10:34:39.381296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:39.381269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:39.381698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:39.381335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:39.393293 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:39.393271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:39.432903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:39.432880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4"
Sep 14 10:34:42.201159 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:34:42.201131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:35:31.123006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:31.122971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:35:52.193365 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.193281    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76b76478db-s82q9"]
Sep 14 10:35:52.196145 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.196118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.211804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.211780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76b76478db-s82q9"]
Sep 14 10:35:52.242407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.242379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqxj\" (UniqueName: \"kubernetes.io/projected/5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84-kube-api-access-dtqxj\") pod \"authorino-76b76478db-s82q9\" (UID: \"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84\") " pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.242516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.242410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84-tls-cert\") pod \"authorino-76b76478db-s82q9\" (UID: \"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84\") " pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.343603 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.343552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqxj\" (UniqueName: \"kubernetes.io/projected/5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84-kube-api-access-dtqxj\") pod \"authorino-76b76478db-s82q9\" (UID: \"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84\") " pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.343711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.343614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84-tls-cert\") pod \"authorino-76b76478db-s82q9\" (UID: \"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84\") " pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.346103 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.346080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84-tls-cert\") pod \"authorino-76b76478db-s82q9\" (UID: \"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84\") " pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.352030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.352008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqxj\" (UniqueName: \"kubernetes.io/projected/5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84-kube-api-access-dtqxj\") pod \"authorino-76b76478db-s82q9\" (UID: \"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84\") " pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.505265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.505190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76b76478db-s82q9"
Sep 14 10:35:52.775027 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:52.774967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76b76478db-s82q9"]
Sep 14 10:35:52.777712 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:35:52.777682    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f14aba0_3dcf_4a09_ab1a_ca8f7cd4fd84.slice/crio-e7892368d06455004dc6aff4cc326ea718b7c84286a3fd8ef46b7504cf1043af WatchSource:0}: Error finding container e7892368d06455004dc6aff4cc326ea718b7c84286a3fd8ef46b7504cf1043af: Status 404 returned error can't find the container with id e7892368d06455004dc6aff4cc326ea718b7c84286a3fd8ef46b7504cf1043af
Sep 14 10:35:53.762098 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:53.762062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76b76478db-s82q9" event={"ID":"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84","Type":"ContainerStarted","Data":"f38788e62ffd90bb6c02e5bbe357af1afc5d3d177508f0cb0f0206768ad2725b"}
Sep 14 10:35:53.762533 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:53.762115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76b76478db-s82q9" event={"ID":"5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84","Type":"ContainerStarted","Data":"e7892368d06455004dc6aff4cc326ea718b7c84286a3fd8ef46b7504cf1043af"}
Sep 14 10:35:53.780817 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:53.780759    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76b76478db-s82q9" podStartSLOduration=0.914614978 podStartE2EDuration="1.780737921s" totalImagesPullingTime="866.122943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:35:53.780189631 +0000 UTC m=+642.335377555" watchObservedRunningTime="2026-09-14 10:35:53.780737921 +0000 UTC m=+642.335925849"
Sep 14 10:35:53.809528 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:53.809495    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d94c75687-jpfpj"]
Sep 14 10:35:53.809744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:53.809719    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d94c75687-jpfpj" podUID="e96551db-f075-49b9-8b6f-1e205b60740e" containerName="authorino" containerID="cri-o://b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83" gracePeriod=30
Sep 14 10:35:54.042117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.042093    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:35:54.058728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.058684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e96551db-f075-49b9-8b6f-1e205b60740e-tls-cert\") pod \"e96551db-f075-49b9-8b6f-1e205b60740e\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") "
Sep 14 10:35:54.058919 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.058810    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bgjn\" (UniqueName: \"kubernetes.io/projected/e96551db-f075-49b9-8b6f-1e205b60740e-kube-api-access-4bgjn\") pod \"e96551db-f075-49b9-8b6f-1e205b60740e\" (UID: \"e96551db-f075-49b9-8b6f-1e205b60740e\") "
Sep 14 10:35:54.061718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.061182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96551db-f075-49b9-8b6f-1e205b60740e-kube-api-access-4bgjn" (OuterVolumeSpecName: "kube-api-access-4bgjn") pod "e96551db-f075-49b9-8b6f-1e205b60740e" (UID: "e96551db-f075-49b9-8b6f-1e205b60740e"). InnerVolumeSpecName "kube-api-access-4bgjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:35:54.070382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.070348    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e96551db-f075-49b9-8b6f-1e205b60740e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e96551db-f075-49b9-8b6f-1e205b60740e" (UID: "e96551db-f075-49b9-8b6f-1e205b60740e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:35:54.159717 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.159682    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e96551db-f075-49b9-8b6f-1e205b60740e-tls-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:35:54.159717 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.159710    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bgjn\" (UniqueName: \"kubernetes.io/projected/e96551db-f075-49b9-8b6f-1e205b60740e-kube-api-access-4bgjn\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:35:54.352148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.352071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:35:54.444458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.444418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:35:54.770355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.770319    2563 generic.go:356] "Generic (PLEG): container finished" podID="e96551db-f075-49b9-8b6f-1e205b60740e" containerID="b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83" exitCode=0
Sep 14 10:35:54.770803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.770370    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d94c75687-jpfpj"
Sep 14 10:35:54.770803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.770422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d94c75687-jpfpj" event={"ID":"e96551db-f075-49b9-8b6f-1e205b60740e","Type":"ContainerDied","Data":"b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83"}
Sep 14 10:35:54.770803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.770490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d94c75687-jpfpj" event={"ID":"e96551db-f075-49b9-8b6f-1e205b60740e","Type":"ContainerDied","Data":"aca473743d8c378ce99d506e7afe5bd5a18d480ae19d6c810b9d5fad6a7bdac9"}
Sep 14 10:35:54.770803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.770516    2563 scope.go:117] "RemoveContainer" containerID="b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83"
Sep 14 10:35:54.779505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.779480    2563 scope.go:117] "RemoveContainer" containerID="b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83"
Sep 14 10:35:54.779863 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:35:54.779844    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83\": container with ID starting with b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83 not found: ID does not exist" containerID="b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83"
Sep 14 10:35:54.779921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.779873    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83"} err="failed to get container status \"b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83\": rpc error: code = NotFound desc = could not find container \"b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83\": container with ID starting with b40b5d7e3d4bdf8979bb276b6601f046c3d28eda624c896e965d6dba3aaa9d83 not found: ID does not exist"
Sep 14 10:35:54.790579 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.790535    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d94c75687-jpfpj"]
Sep 14 10:35:54.794496 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:54.794475    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d94c75687-jpfpj"]
Sep 14 10:35:55.509308 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:55.509280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:35:55.607145 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:55.607113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:35:55.918343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:55.918310    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96551db-f075-49b9-8b6f-1e205b60740e" path="/var/lib/kubelet/pods/e96551db-f075-49b9-8b6f-1e205b60740e/volumes"
Sep 14 10:35:59.420124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:59.420080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:35:59.518500 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:35:59.518467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.039447 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.039406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.139763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.139729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.248203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.248168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.412972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.412942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.539967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.539933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.626115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.626083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.699947 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.699842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.800939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.800905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:13.873521 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:13.873488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:14.574804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:14.574762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:14.666513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:14.666479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:15.072792 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.072755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:15.106881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.106852    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7738-d5dac0"]
Sep 14 10:36:15.107200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.107187    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96551db-f075-49b9-8b6f-1e205b60740e" containerName="authorino"
Sep 14 10:36:15.107247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.107201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96551db-f075-49b9-8b6f-1e205b60740e" containerName="authorino"
Sep 14 10:36:15.107293 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.107268    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96551db-f075-49b9-8b6f-1e205b60740e" containerName="authorino"
Sep 14 10:36:15.110011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.109994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:15.112407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.112388    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pp8dj\""
Sep 14 10:36:15.115635 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.115610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7738-d5dac0"]
Sep 14 10:36:15.242583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.242543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44nzx\" (UniqueName: \"kubernetes.io/projected/b36a45ac-a687-4efa-a697-72d442c948cb-kube-api-access-44nzx\") pod \"test-curl-7738-d5dac0\" (UID: \"b36a45ac-a687-4efa-a697-72d442c948cb\") " pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:15.343445 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.343370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44nzx\" (UniqueName: \"kubernetes.io/projected/b36a45ac-a687-4efa-a697-72d442c948cb-kube-api-access-44nzx\") pod \"test-curl-7738-d5dac0\" (UID: \"b36a45ac-a687-4efa-a697-72d442c948cb\") " pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:15.353606 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.353554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44nzx\" (UniqueName: \"kubernetes.io/projected/b36a45ac-a687-4efa-a697-72d442c948cb-kube-api-access-44nzx\") pod \"test-curl-7738-d5dac0\" (UID: \"b36a45ac-a687-4efa-a697-72d442c948cb\") " pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:15.419973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.419938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:15.691115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.691086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7738-d5dac0"]
Sep 14 10:36:15.693346 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:15.693321    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb36a45ac_a687_4efa_a697_72d442c948cb.slice/crio-7d0d1536297ed95a77ada78b8a78752d0c0418644024c7f9ec2c17e6b7d16a73 WatchSource:0}: Error finding container 7d0d1536297ed95a77ada78b8a78752d0c0418644024c7f9ec2c17e6b7d16a73: Status 404 returned error can't find the container with id 7d0d1536297ed95a77ada78b8a78752d0c0418644024c7f9ec2c17e6b7d16a73
Sep 14 10:36:15.695383 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.695365    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:36:15.867337 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:15.867303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7738-d5dac0" event={"ID":"b36a45ac-a687-4efa-a697-72d442c948cb","Type":"ContainerStarted","Data":"7d0d1536297ed95a77ada78b8a78752d0c0418644024c7f9ec2c17e6b7d16a73"}
Sep 14 10:36:16.551770 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:16.551733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:16.692372 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:16.692341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:16.776538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:16.776476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:17.369684 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:17.369647    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:36:17.906298 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:17.904897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7738-d5dac0" event={"ID":"b36a45ac-a687-4efa-a697-72d442c948cb","Type":"ContainerStarted","Data":"6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239"}
Sep 14 10:36:17.926388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:17.926336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7738-d5dac0" podStartSLOduration=0.831912728 podStartE2EDuration="2.926320226s" totalImagesPullingTime="2.094407498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:17.923721187 +0000 UTC m=+666.478909128" watchObservedRunningTime="2026-09-14 10:36:17.926320226 +0000 UTC m=+666.481508152"
Sep 14 10:36:18.516511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.516472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:18.532947 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.532909    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7738-d5dac0"]
Sep 14 10:36:18.539650 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.539615    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7738-d5dac0"]
Sep 14 10:36:18.616077 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.615985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:18.658827 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.658795    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7738-7c3cd1"]
Sep 14 10:36:18.659150 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.659138    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" containerName="test-curl-7738-d5dac0"
Sep 14 10:36:18.659196 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.659152    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" containerName="test-curl-7738-d5dac0"
Sep 14 10:36:18.659229 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.659207    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" containerName="test-curl-7738-d5dac0"
Sep 14 10:36:18.661336 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.661320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:18.666658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.666635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7738-7c3cd1"]
Sep 14 10:36:18.775658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.775625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952mr\" (UniqueName: \"kubernetes.io/projected/ae1f0b9c-f238-4333-bdb6-34e1719ead19-kube-api-access-952mr\") pod \"test-curl-7738-7c3cd1\" (UID: \"ae1f0b9c-f238-4333-bdb6-34e1719ead19\") " pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:18.799115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.799083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:18.876256 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.876220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952mr\" (UniqueName: \"kubernetes.io/projected/ae1f0b9c-f238-4333-bdb6-34e1719ead19-kube-api-access-952mr\") pod \"test-curl-7738-7c3cd1\" (UID: \"ae1f0b9c-f238-4333-bdb6-34e1719ead19\") " pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:18.884265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.884238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-952mr\" (UniqueName: \"kubernetes.io/projected/ae1f0b9c-f238-4333-bdb6-34e1719ead19-kube-api-access-952mr\") pod \"test-curl-7738-7c3cd1\" (UID: \"ae1f0b9c-f238-4333-bdb6-34e1719ead19\") " pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:18.909694 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:18.909665    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7738-d5dac0" container="test-curl-7738-d5dac0"
Sep 14 10:36:18.970839 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:18.970807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:19.003431 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.003399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:19.118507 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.118477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:19.241429 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.241344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7738-7c3cd1"]
Sep 14 10:36:19.244101 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:19.244070    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae1f0b9c_f238_4333_bdb6_34e1719ead19.slice/crio-95a58b4ab5708efb558a1898bf5eed985a3642626172b3b4a0183ba6243e1cf8 WatchSource:0}: Error finding container 95a58b4ab5708efb558a1898bf5eed985a3642626172b3b4a0183ba6243e1cf8: Status 404 returned error can't find the container with id 95a58b4ab5708efb558a1898bf5eed985a3642626172b3b4a0183ba6243e1cf8
Sep 14 10:36:19.429324 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.429288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:19.493148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.493066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:19.712340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.712290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:19.916433 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.916396    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7738-d5dac0" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" containerName="test-curl-7738-d5dac0" containerID="cri-o://6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239" gracePeriod=2
Sep 14 10:36:19.919182 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.919158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7738-7c3cd1" event={"ID":"ae1f0b9c-f238-4333-bdb6-34e1719ead19","Type":"ContainerStarted","Data":"5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d"}
Sep 14 10:36:19.919281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.919195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7738-7c3cd1" event={"ID":"ae1f0b9c-f238-4333-bdb6-34e1719ead19","Type":"ContainerStarted","Data":"95a58b4ab5708efb558a1898bf5eed985a3642626172b3b4a0183ba6243e1cf8"}
Sep 14 10:36:19.929933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.929904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:19.941832 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:19.941780    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7738-7c3cd1" podStartSLOduration=1.941762755 podStartE2EDuration="1.941762755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:19.940867616 +0000 UTC m=+668.496055542" watchObservedRunningTime="2026-09-14 10:36:19.941762755 +0000 UTC m=+668.496950685"
Sep 14 10:36:20.005308 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.005274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:20.027156 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.027128    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"]
Sep 14 10:36:20.029447 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.029430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.032666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.032637    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-85d3ec-dockercfg-qfwq8\""
Sep 14 10:36:20.032666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.032655    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-85d3ec\""
Sep 14 10:36:20.044840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.044801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"]
Sep 14 10:36:20.047536 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.047507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"]
Sep 14 10:36:20.047675 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.047649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.061407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.061380    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"]
Sep 14 10:36:20.187377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.187298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbdff61-8069-4450-909a-d747f5f6f9bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.187377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.187347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mp8\" (UniqueName: \"kubernetes.io/projected/adbdff61-8069-4450-909a-d747f5f6f9bb-kube-api-access-n7mp8\") pod \"payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.187600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.187392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da661d50-70df-43bb-866f-ac1a4907c1e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.187600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.187442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdft4\" (UniqueName: \"kubernetes.io/projected/da661d50-70df-43bb-866f-ac1a4907c1e0-kube-api-access-pdft4\") pod \"payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.288562 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.288520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da661d50-70df-43bb-866f-ac1a4907c1e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.288769 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.288596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdft4\" (UniqueName: \"kubernetes.io/projected/da661d50-70df-43bb-866f-ac1a4907c1e0-kube-api-access-pdft4\") pod \"payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.288769 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.288666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbdff61-8069-4450-909a-d747f5f6f9bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.288769 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.288695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mp8\" (UniqueName: \"kubernetes.io/projected/adbdff61-8069-4450-909a-d747f5f6f9bb-kube-api-access-n7mp8\") pod \"payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.289451 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.289426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbdff61-8069-4450-909a-d747f5f6f9bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.289525 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.289426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da661d50-70df-43bb-866f-ac1a4907c1e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.297331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.297303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdft4\" (UniqueName: \"kubernetes.io/projected/da661d50-70df-43bb-866f-ac1a4907c1e0-kube-api-access-pdft4\") pod \"payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.297331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.297323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mp8\" (UniqueName: \"kubernetes.io/projected/adbdff61-8069-4450-909a-d747f5f6f9bb-kube-api-access-n7mp8\") pod \"payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.339355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.339319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:20.361197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.361169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:20.542121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.542035    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7738-7c3cd1"]
Sep 14 10:36:20.547798 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.547771    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7738-7c3cd1"]
Sep 14 10:36:20.595941 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.595909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:20.629684 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.629659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"]
Sep 14 10:36:20.631721 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:20.631692    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda661d50_70df_43bb_866f_ac1a4907c1e0.slice/crio-275dd6bafacdac55d74a9954e643aea25d65660d6d7eb72caf8b41162ab757e8 WatchSource:0}: Error finding container 275dd6bafacdac55d74a9954e643aea25d65660d6d7eb72caf8b41162ab757e8: Status 404 returned error can't find the container with id 275dd6bafacdac55d74a9954e643aea25d65660d6d7eb72caf8b41162ab757e8
Sep 14 10:36:20.652782 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.652757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"]
Sep 14 10:36:20.654704 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:20.654677    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbdff61_8069_4450_909a_d747f5f6f9bb.slice/crio-a1dfb23cd985a79ac30a8697619b4341d25a0c1cb6b88bafe472aaf8e27d460b WatchSource:0}: Error finding container a1dfb23cd985a79ac30a8697619b4341d25a0c1cb6b88bafe472aaf8e27d460b: Status 404 returned error can't find the container with id a1dfb23cd985a79ac30a8697619b4341d25a0c1cb6b88bafe472aaf8e27d460b
Sep 14 10:36:20.921699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.921665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp" event={"ID":"da661d50-70df-43bb-866f-ac1a4907c1e0","Type":"ContainerStarted","Data":"275dd6bafacdac55d74a9954e643aea25d65660d6d7eb72caf8b41162ab757e8"}
Sep 14 10:36:20.922908 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:20.922883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h" event={"ID":"adbdff61-8069-4450-909a-d747f5f6f9bb","Type":"ContainerStarted","Data":"a1dfb23cd985a79ac30a8697619b4341d25a0c1cb6b88bafe472aaf8e27d460b"}
Sep 14 10:36:20.923076 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:20.923057    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7738-7c3cd1" container="test-curl-7738-7c3cd1"
Sep 14 10:36:21.091849 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:21.091813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:21.230038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:21.229960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:21.367430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:21.367395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:21.552304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:21.548048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:21.923208 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:21.922765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:21.927172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:21.927143    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7738-7c3cd1" podUID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" containerName="test-curl-7738-7c3cd1" containerID="cri-o://5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d" gracePeriod=2
Sep 14 10:36:22.104454 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.104428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:22.106264 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:22.106234    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-d5dac0\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:22.123838 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.123812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:22.204959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.204864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44nzx\" (UniqueName: \"kubernetes.io/projected/b36a45ac-a687-4efa-a697-72d442c948cb-kube-api-access-44nzx\") pod \"b36a45ac-a687-4efa-a697-72d442c948cb\" (UID: \"b36a45ac-a687-4efa-a697-72d442c948cb\") "
Sep 14 10:36:22.207252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.207228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36a45ac-a687-4efa-a697-72d442c948cb-kube-api-access-44nzx" (OuterVolumeSpecName: "kube-api-access-44nzx") pod "b36a45ac-a687-4efa-a697-72d442c948cb" (UID: "b36a45ac-a687-4efa-a697-72d442c948cb"). InnerVolumeSpecName "kube-api-access-44nzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:36:22.305794 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.305761    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44nzx\" (UniqueName: \"kubernetes.io/projected/b36a45ac-a687-4efa-a697-72d442c948cb-kube-api-access-44nzx\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:36:22.584149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.584056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:22.683680 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.683640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:22.791247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.791208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:22.933890 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.933844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h" event={"ID":"adbdff61-8069-4450-909a-d747f5f6f9bb","Type":"ContainerStarted","Data":"a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699"}
Sep 14 10:36:22.934369 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.934153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:22.935225 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.935194    2563 generic.go:356] "Generic (PLEG): container finished" podID="b36a45ac-a687-4efa-a697-72d442c948cb" containerID="6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239" exitCode=137
Sep 14 10:36:22.935351 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.935251    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:22.935351 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.935270    2563 scope.go:117] "RemoveContainer" containerID="6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239"
Sep 14 10:36:22.939667 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:22.939326    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-d5dac0\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:22.940901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.940882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:22.941439 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.941413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp" event={"ID":"da661d50-70df-43bb-866f-ac1a4907c1e0","Type":"ContainerStarted","Data":"47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee"}
Sep 14 10:36:22.941646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.941631    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:22.948684 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.948651    2563 scope.go:117] "RemoveContainer" containerID="6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239"
Sep 14 10:36:22.948936 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:22.948915    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239\": container with ID starting with 6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239 not found: ID does not exist" containerID="6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239"
Sep 14 10:36:22.948997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.948943    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239"} err="failed to get container status \"6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239\": rpc error: code = NotFound desc = could not find container \"6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239\": container with ID starting with 6a26dc61f473a8838cf380ef08fd29bfa5f3ce5c9bcd51ccd24e73cf208d2239 not found: ID does not exist"
Sep 14 10:36:22.983765 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:22.983717    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h" podStartSLOduration=1.691278752 podStartE2EDuration="2.983702781s" totalImagesPullingTime="1.292424029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:22.982742068 +0000 UTC m=+671.537929993" watchObservedRunningTime="2026-09-14 10:36:22.983702781 +0000 UTC m=+671.538890706"
Sep 14 10:36:23.027269 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.027211    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp" podStartSLOduration=1.7097892190000001 podStartE2EDuration="3.02719778s" totalImagesPullingTime="1.317408561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:23.02587248 +0000 UTC m=+671.581060429" watchObservedRunningTime="2026-09-14 10:36:23.02719778 +0000 UTC m=+671.582385705"
Sep 14 10:36:23.027436 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:23.027334    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-d5dac0\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:23.201972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.201901    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"]
Sep 14 10:36:23.202297 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.202284    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" containerName="test-curl-7738-7c3cd1"
Sep 14 10:36:23.202344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.202299    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" containerName="test-curl-7738-7c3cd1"
Sep 14 10:36:23.202379 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.202371    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" containerName="test-curl-7738-7c3cd1"
Sep 14 10:36:23.206793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.206772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.212623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.212600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 10:36:23.212737 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.212617    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3be523b2-openshift-default-dockercfg-6fhc9\""
Sep 14 10:36:23.212908 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:23.212882    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-d5dac0\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" pod="opendatahub/test-curl-7738-d5dac0"
Sep 14 10:36:23.230478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.230450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"]
Sep 14 10:36:23.261682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.261652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:23.315994 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.315950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-certs\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-data\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-token\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istiod-ca-cert\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-podinfo\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-envoy\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-socket\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-credential-socket\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.316430 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.316279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9kfx\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-kube-api-access-s9kfx\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.416789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.416745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-podinfo\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.416789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.416790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-envoy\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.417149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.416837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-socket\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.417149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.416872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-credential-socket\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.417149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.416909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9kfx\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-kube-api-access-s9kfx\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.417149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.416965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-certs\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.417149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.416987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-data\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.417149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.417026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-token\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.417149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.417049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istiod-ca-cert\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.418333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.417607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-certs\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.418333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.417627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-credential-socket\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.418333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.417684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-data\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.418333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.417915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-socket\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.419281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.419249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istiod-ca-cert\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.419669 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.419646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-envoy\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.419782 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.419759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-podinfo\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.426853 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.426832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-token\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.427392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.427371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9kfx\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-kube-api-access-s9kfx\") pod \"e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") " pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.441371 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.441344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:23.518132 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.518040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:23.796525 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.796452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"]
Sep 14 10:36:23.799835 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:23.799808    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96300b2_8222_4023_a4cf_07ba8c5c68f2.slice/crio-c95ce91e640d07791ee1a86c7ccb1a0a9c9c1162e68901faf05d9ce2e676c0d2 WatchSource:0}: Error finding container c95ce91e640d07791ee1a86c7ccb1a0a9c9c1162e68901faf05d9ce2e676c0d2: Status 404 returned error can't find the container with id c95ce91e640d07791ee1a86c7ccb1a0a9c9c1162e68901faf05d9ce2e676c0d2
Sep 14 10:36:23.901122 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.901087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:23.918910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.918876    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36a45ac-a687-4efa-a697-72d442c948cb" path="/var/lib/kubelet/pods/b36a45ac-a687-4efa-a697-72d442c948cb/volumes"
Sep 14 10:36:23.948665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:23.948627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" event={"ID":"a96300b2-8222-4023-a4cf-07ba8c5c68f2","Type":"ContainerStarted","Data":"c95ce91e640d07791ee1a86c7ccb1a0a9c9c1162e68901faf05d9ce2e676c0d2"}
Sep 14 10:36:24.080300 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.080276    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:24.225132 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.225099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-952mr\" (UniqueName: \"kubernetes.io/projected/ae1f0b9c-f238-4333-bdb6-34e1719ead19-kube-api-access-952mr\") pod \"ae1f0b9c-f238-4333-bdb6-34e1719ead19\" (UID: \"ae1f0b9c-f238-4333-bdb6-34e1719ead19\") "
Sep 14 10:36:24.227336 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.227304    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1f0b9c-f238-4333-bdb6-34e1719ead19-kube-api-access-952mr" (OuterVolumeSpecName: "kube-api-access-952mr") pod "ae1f0b9c-f238-4333-bdb6-34e1719ead19" (UID: "ae1f0b9c-f238-4333-bdb6-34e1719ead19"). InnerVolumeSpecName "kube-api-access-952mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:36:24.326379 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.326299    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952mr\" (UniqueName: \"kubernetes.io/projected/ae1f0b9c-f238-4333-bdb6-34e1719ead19-kube-api-access-952mr\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:36:24.840921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.840869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:24.955224 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.955185    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" containerID="5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d" exitCode=137
Sep 14 10:36:24.955701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.955243    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:24.955701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.955282    2563 scope.go:117] "RemoveContainer" containerID="5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d"
Sep 14 10:36:24.963737 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.963544    2563 scope.go:117] "RemoveContainer" containerID="5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d"
Sep 14 10:36:24.963846 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:24.963826    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d\": container with ID starting with 5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d not found: ID does not exist" containerID="5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d"
Sep 14 10:36:24.963879 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:24.963853    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d"} err="failed to get container status \"5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d\": rpc error: code = NotFound desc = could not find container \"5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d\": container with ID starting with 5db7fdf129c7f6f5e38ff26e303a4d8baa5e2713066adb03c12c53a7afb2e72d not found: ID does not exist"
Sep 14 10:36:24.965246 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:24.965226    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-7c3cd1\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object" podUID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:25.011813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.011771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:25.196393 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.196343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:25.612304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.612194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:25.706322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.706267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:25.746635 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.746564    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"]
Sep 14 10:36:25.750026 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.750005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:25.751786 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:36:25.751756    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-7c3cd1\" is forbidden: User \"system:node:ip-10-0-133-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-125.ec2.internal' and this object" podUID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" pod="opendatahub/test-curl-7738-7c3cd1"
Sep 14 10:36:25.752029 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.752007    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-1910e9\""
Sep 14 10:36:25.752279 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.752263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-1910e9-dockercfg-mqrd7\""
Sep 14 10:36:25.765661 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.765624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"]
Sep 14 10:36:25.839768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.839726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9515cd-3937-48b8-b230-52a9d74fbb63-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:25.839958 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.839799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnv9\" (UniqueName: \"kubernetes.io/projected/8b9515cd-3937-48b8-b230-52a9d74fbb63-kube-api-access-ltnv9\") pod \"payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:25.920513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.920476    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1f0b9c-f238-4333-bdb6-34e1719ead19" path="/var/lib/kubelet/pods/ae1f0b9c-f238-4333-bdb6-34e1719ead19/volumes"
Sep 14 10:36:25.940347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.940307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnv9\" (UniqueName: \"kubernetes.io/projected/8b9515cd-3937-48b8-b230-52a9d74fbb63-kube-api-access-ltnv9\") pod \"payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:25.940525 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.940413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9515cd-3937-48b8-b230-52a9d74fbb63-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:25.941237 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.941208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9515cd-3937-48b8-b230-52a9d74fbb63-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:25.948531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:25.948500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnv9\" (UniqueName: \"kubernetes.io/projected/8b9515cd-3937-48b8-b230-52a9d74fbb63-kube-api-access-ltnv9\") pod \"payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:26.060965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:26.060925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:26.261960 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:26.261875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:26.347114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:26.347083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"]
Sep 14 10:36:26.349505 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:26.349472    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9515cd_3937_48b8_b230_52a9d74fbb63.slice/crio-7e948504b44929791e96caa4fff4a581d9ce4af1c2b9b7cde71aaa39b0c0681b WatchSource:0}: Error finding container 7e948504b44929791e96caa4fff4a581d9ce4af1c2b9b7cde71aaa39b0c0681b: Status 404 returned error can't find the container with id 7e948504b44929791e96caa4fff4a581d9ce4af1c2b9b7cde71aaa39b0c0681b
Sep 14 10:36:26.971162 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:26.971120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh" event={"ID":"8b9515cd-3937-48b8-b230-52a9d74fbb63","Type":"ContainerStarted","Data":"6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b"}
Sep 14 10:36:26.971346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:26.971187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh" event={"ID":"8b9515cd-3937-48b8-b230-52a9d74fbb63","Type":"ContainerStarted","Data":"7e948504b44929791e96caa4fff4a581d9ce4af1c2b9b7cde71aaa39b0c0681b"}
Sep 14 10:36:26.971346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:26.971216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:27.211022 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.210987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:27.319455 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.319369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:27.753456 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.753416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:27.861556 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.861523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:27.936845 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.936806    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:36:27.936944 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.936897    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:36:27.936944 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.936938    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:36:27.954969 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:27.954944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:28.029445 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.029397    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh" podStartSLOduration=3.029378925 podStartE2EDuration="3.029378925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:26.987849829 +0000 UTC m=+675.543037755" watchObservedRunningTime="2026-09-14 10:36:28.029378925 +0000 UTC m=+676.584566881"
Sep 14 10:36:28.030712 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.030687    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"]
Sep 14 10:36:28.034597 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.034552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.036665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.036645    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-30eaba-dockercfg-6kr88\""
Sep 14 10:36:28.036770 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.036694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-30eaba\""
Sep 14 10:36:28.044934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.044908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"]
Sep 14 10:36:28.163743 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.163700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpgw\" (UniqueName: \"kubernetes.io/projected/b0f383d4-f629-4641-89c7-8fd696d15372-kube-api-access-bbpgw\") pod \"payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.163919 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.163798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f383d4-f629-4641-89c7-8fd696d15372-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.264842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.264764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpgw\" (UniqueName: \"kubernetes.io/projected/b0f383d4-f629-4641-89c7-8fd696d15372-kube-api-access-bbpgw\") pod \"payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.264842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.264830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f383d4-f629-4641-89c7-8fd696d15372-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.265464 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.265446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f383d4-f629-4641-89c7-8fd696d15372-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.273602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.273564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpgw\" (UniqueName: \"kubernetes.io/projected/b0f383d4-f629-4641-89c7-8fd696d15372-kube-api-access-bbpgw\") pod \"payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.346611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.346563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:28.591503 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.591421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:28.625612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.623595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"]
Sep 14 10:36:28.982813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.982772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" event={"ID":"a96300b2-8222-4023-a4cf-07ba8c5c68f2","Type":"ContainerStarted","Data":"283b5657b735eb46a7f7cf700b4d78ace5e861ec98590e0c50df32a4e88b7dae"}
Sep 14 10:36:28.984376 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.984351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf" event={"ID":"b0f383d4-f629-4641-89c7-8fd696d15372","Type":"ContainerStarted","Data":"d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038"}
Sep 14 10:36:28.984495 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.984395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf" event={"ID":"b0f383d4-f629-4641-89c7-8fd696d15372","Type":"ContainerStarted","Data":"51bcc6fd8713eb19f58a94476f6a9bda16967ac52631fb8ee88c2d958f9db8e2"}
Sep 14 10:36:28.984495 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:28.984486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:29.002248 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:29.002197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" podStartSLOduration=1.8678562749999998 podStartE2EDuration="6.002178624s" totalImagesPullingTime="4.134322349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:29.000908836 +0000 UTC m=+677.556096781" watchObservedRunningTime="2026-09-14 10:36:29.002178624 +0000 UTC m=+677.557366552"
Sep 14 10:36:29.017403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:29.017357    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf" podStartSLOduration=1.017343486 podStartE2EDuration="1.017343486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:29.01645978 +0000 UTC m=+677.571647706" watchObservedRunningTime="2026-09-14 10:36:29.017343486 +0000 UTC m=+677.572531411"
Sep 14 10:36:29.518865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:29.518826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:29.524204 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:29.524179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:29.557255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:29.557227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:29.989352 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:29.989315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:29.990199 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:29.990176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:30.775957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:30.775923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:30.873052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:30.873018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:30.973702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:30.973670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:31.321021 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:31.320949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:31.417977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:31.417941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:31.726407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:31.726374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:31.914252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:31.914211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:32.111503 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:32.111405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:32.202320 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:32.202278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:32.623296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:32.623267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:33.221326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:33.221284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:33.545240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:33.545146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:33.951740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:33.951709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:36:33.951899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:33.951775    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:36:33.973553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:33.973523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:34.174534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:34.174499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:34.256332 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:34.256247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:34.353357 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:34.353319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:34.903361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:34.903333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:35.007111 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.007075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:35.343634 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.343522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:35.445010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.444961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:35.524287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.524253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:35.525017 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.524997    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"]
Sep 14 10:36:35.528032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.528015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.530463 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.530447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3be523b2\""
Sep 14 10:36:35.533641 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.533625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3be523b2-dockercfg-kcdxh\""
Sep 14 10:36:35.543743 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.543724    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"]
Sep 14 10:36:35.627168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.627128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36e3a8d-588b-4cc4-b391-205d49b5dde2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.627168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.627168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7pw\" (UniqueName: \"kubernetes.io/projected/f36e3a8d-588b-4cc4-b391-205d49b5dde2-kube-api-access-5t7pw\") pod \"payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.728239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.728209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36e3a8d-588b-4cc4-b391-205d49b5dde2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.728239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.728243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7pw\" (UniqueName: \"kubernetes.io/projected/f36e3a8d-588b-4cc4-b391-205d49b5dde2-kube-api-access-5t7pw\") pod \"payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.728957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.728939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36e3a8d-588b-4cc4-b391-205d49b5dde2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.736240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.736213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7pw\" (UniqueName: \"kubernetes.io/projected/f36e3a8d-588b-4cc4-b391-205d49b5dde2-kube-api-access-5t7pw\") pod \"payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.838884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.838828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:35.991675 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:35.991641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:36.072473 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:36.072437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:36.114713 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:36.114688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"]
Sep 14 10:36:36.116909 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:36.116875    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36e3a8d_588b_4cc4_b391_205d49b5dde2.slice/crio-13a677d871902f50a0b8fdc9ed3b435c7d83d7dda1daf8d74f068ffc77f699a6 WatchSource:0}: Error finding container 13a677d871902f50a0b8fdc9ed3b435c7d83d7dda1daf8d74f068ffc77f699a6: Status 404 returned error can't find the container with id 13a677d871902f50a0b8fdc9ed3b435c7d83d7dda1daf8d74f068ffc77f699a6
Sep 14 10:36:36.152898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:36.152869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:36.355283 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:36.355249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:36.454447 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:36.454370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:37.025636 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:37.025602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm" event={"ID":"f36e3a8d-588b-4cc4-b391-205d49b5dde2","Type":"ContainerStarted","Data":"aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0"}
Sep 14 10:36:37.025803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:37.025660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm" event={"ID":"f36e3a8d-588b-4cc4-b391-205d49b5dde2","Type":"ContainerStarted","Data":"13a677d871902f50a0b8fdc9ed3b435c7d83d7dda1daf8d74f068ffc77f699a6"}
Sep 14 10:36:37.025803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:37.025699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:37.045312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:37.045280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:37.055834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:37.055793    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm" podStartSLOduration=2.055778801 podStartE2EDuration="2.055778801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:37.053639948 +0000 UTC m=+685.608827873" watchObservedRunningTime="2026-09-14 10:36:37.055778801 +0000 UTC m=+685.610966727"
Sep 14 10:36:37.184505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:37.184475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:37.978893 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:37.978862    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:36:38.185914 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:38.185881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:38.847065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:38.847035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:39.090646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:39.090613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:36:39.390893 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:39.390855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:39.491063 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:39.491025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:39.610581 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:39.610535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:39.991773 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:39.991744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:36:40.132824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:40.132777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:40.225478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:40.225442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:40.768247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:40.768200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:40.915375 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:40.915332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:41.379521 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:41.379487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:41.472466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:41.472430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:41.574717 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:41.574688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:42.360395 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:42.360363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:43.780074 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:43.780035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:44.498502 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:44.498462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:45.066200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:45.066152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:46.132355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:46.132314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:47.041067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:47.041035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:47.788020 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:47.787981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:48.033995 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:48.033961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:36:48.673228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:48.673187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:49.485865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:49.485828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:50.319436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:50.319405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:50.589228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:50.589150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:50.948200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:50.948171    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"]
Sep 14 10:36:50.950758 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:50.950742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"
Sep 14 10:36:50.952811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:50.952791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6vmmw\""
Sep 14 10:36:50.952900 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:50.952830    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:36:50.958912 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:50.958892    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"]
Sep 14 10:36:51.016284 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.016254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:51.084542 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.084507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c64b19-5188-4e54-9d89-4e9488a816b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn\" (UID: \"c6c64b19-5188-4e54-9d89-4e9488a816b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"
Sep 14 10:36:51.099231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.099199    2563 generic.go:356] "Generic (PLEG): container finished" podID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerID="283b5657b735eb46a7f7cf700b4d78ace5e861ec98590e0c50df32a4e88b7dae" exitCode=0
Sep 14 10:36:51.099351 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.099273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" event={"ID":"a96300b2-8222-4023-a4cf-07ba8c5c68f2","Type":"ContainerDied","Data":"283b5657b735eb46a7f7cf700b4d78ace5e861ec98590e0c50df32a4e88b7dae"}
Sep 14 10:36:51.099710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.099696    2563 scope.go:117] "RemoveContainer" containerID="283b5657b735eb46a7f7cf700b4d78ace5e861ec98590e0c50df32a4e88b7dae"
Sep 14 10:36:51.100949 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.100923    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:36:51.101058 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.101010    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:36:51.101122 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.101058    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:36:51.185915 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.185878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c64b19-5188-4e54-9d89-4e9488a816b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn\" (UID: \"c6c64b19-5188-4e54-9d89-4e9488a816b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"
Sep 14 10:36:51.186640 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.186612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c64b19-5188-4e54-9d89-4e9488a816b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn\" (UID: \"c6c64b19-5188-4e54-9d89-4e9488a816b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"
Sep 14 10:36:51.262236 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.262151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"
Sep 14 10:36:51.540184 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:51.540100    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"]
Sep 14 10:36:51.542380 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:51.542352    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c64b19_5188_4e54_9d89_4e9488a816b4.slice/crio-1512ec3a4ba88b56466ca9dc67d7db1f13c9f40c91c5c17c9eb3c09a418ca1e1 WatchSource:0}: Error finding container 1512ec3a4ba88b56466ca9dc67d7db1f13c9f40c91c5c17c9eb3c09a418ca1e1: Status 404 returned error can't find the container with id 1512ec3a4ba88b56466ca9dc67d7db1f13c9f40c91c5c17c9eb3c09a418ca1e1
Sep 14 10:36:52.051527 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:52.051474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:52.106461 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:52.106429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" event={"ID":"a96300b2-8222-4023-a4cf-07ba8c5c68f2","Type":"ContainerStarted","Data":"9fd672305faa41c7509a679ca7fcd5a79115e362cc460bcc7d82c193a1a05827"}
Sep 14 10:36:52.107714 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:52.107690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn" event={"ID":"c6c64b19-5188-4e54-9d89-4e9488a816b4","Type":"ContainerStarted","Data":"1512ec3a4ba88b56466ca9dc67d7db1f13c9f40c91c5c17c9eb3c09a418ca1e1"}
Sep 14 10:36:52.518663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:52.518608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:52.518877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:52.518751    2563 patch_prober.go:28] interesting pod/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 14 10:36:52.518877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:52.518795    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 14 10:36:53.026924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:53.026882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:53.529596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:53.528981    2563 patch_prober.go:28] interesting pod/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 14 10:36:53.529596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:53.529046    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 14 10:36:53.936993 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:53.936946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:54.528599 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:54.525807    2563 patch_prober.go:28] interesting pod/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 14 10:36:54.528599 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:54.525868    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 14 10:36:54.835188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:54.835094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:55.519084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.518651    2563 patch_prober.go:28] interesting pod/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 14 10:36:55.519084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.518710    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 14 10:36:55.722131 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.722090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:55.990034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.990004    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"]
Sep 14 10:36:55.992621 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.992597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:55.994747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.994721    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-85d3ec\"/\"simulator-w0-85d3ec-kserve-self-signed-certs\""
Sep 14 10:36:55.995232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.995214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-85d3ec\"/\"kube-root-ca.crt\""
Sep 14 10:36:55.995285 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.995238    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-85d3ec\"/\"openshift-service-ca.crt\""
Sep 14 10:36:55.995335 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:55.995302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-85d3ec\"/\"default-dockercfg-wndsg\""
Sep 14 10:36:56.002206 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.002185    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"]
Sep 14 10:36:56.149064 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.149022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn" event={"ID":"c6c64b19-5188-4e54-9d89-4e9488a816b4","Type":"ContainerStarted","Data":"1d868b31112f1632bad21308c91965b880251a7e7dfc2a3c859f586da9aaab76"}
Sep 14 10:36:56.153370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.153344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde1a739-f735-41d6-ac3e-5c01abba71fc-tls-certs\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.153498 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.153395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-model-cache\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.153498 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.153444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvxm\" (UniqueName: \"kubernetes.io/projected/cde1a739-f735-41d6-ac3e-5c01abba71fc-kube-api-access-rdvxm\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.153498 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.153468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-dshm\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.153622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.153546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-home\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.153622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.153596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-tmp-dir\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.164876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.164829    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn" podStartSLOduration=2.511109553 podStartE2EDuration="6.164816084s" totalImagesPullingTime="3.653706531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:56.162732711 +0000 UTC m=+704.717920650" watchObservedRunningTime="2026-09-14 10:36:56.164816084 +0000 UTC m=+704.720004009"
Sep 14 10:36:56.254952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.254860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-model-cache\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.254953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvxm\" (UniqueName: \"kubernetes.io/projected/cde1a739-f735-41d6-ac3e-5c01abba71fc-kube-api-access-rdvxm\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.254982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-dshm\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.255040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-home\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.255055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-tmp-dir\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.255073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde1a739-f735-41d6-ac3e-5c01abba71fc-tls-certs\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.255271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-model-cache\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.255355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-tmp-dir\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.255683 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.255660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-home\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.257399 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.257365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-dshm\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.257944 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.257922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde1a739-f735-41d6-ac3e-5c01abba71fc-tls-certs\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.263834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.263809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvxm\" (UniqueName: \"kubernetes.io/projected/cde1a739-f735-41d6-ac3e-5c01abba71fc-kube-api-access-rdvxm\") pod \"simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") " pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.271438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.271376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:56.302819 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.302795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:36:56.527656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.519989    2563 patch_prober.go:28] interesting pod/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 14 10:36:56.527656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.520049    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 14 10:36:56.600699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.600608    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"]
Sep 14 10:36:56.602951 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:36:56.602922    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcde1a739_f735_41d6_ac3e_5c01abba71fc.slice/crio-8d8087491770d3e1455db17c431760e3666ba84c020198f68249b2d93e725228 WatchSource:0}: Error finding container 8d8087491770d3e1455db17c431760e3666ba84c020198f68249b2d93e725228: Status 404 returned error can't find the container with id 8d8087491770d3e1455db17c431760e3666ba84c020198f68249b2d93e725228
Sep 14 10:36:56.807480 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:56.807397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:57.162158 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.162122    2563 generic.go:356] "Generic (PLEG): container finished" podID="c6c64b19-5188-4e54-9d89-4e9488a816b4" containerID="1d868b31112f1632bad21308c91965b880251a7e7dfc2a3c859f586da9aaab76" exitCode=0
Sep 14 10:36:57.162520 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.162196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn" event={"ID":"c6c64b19-5188-4e54-9d89-4e9488a816b4","Type":"ContainerDied","Data":"1d868b31112f1632bad21308c91965b880251a7e7dfc2a3c859f586da9aaab76"}
Sep 14 10:36:57.163634 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.163606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m" event={"ID":"cde1a739-f735-41d6-ac3e-5c01abba71fc","Type":"ContainerStarted","Data":"9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b"}
Sep 14 10:36:57.163754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.163643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m" event={"ID":"cde1a739-f735-41d6-ac3e-5c01abba71fc","Type":"ContainerStarted","Data":"8d8087491770d3e1455db17c431760e3666ba84c020198f68249b2d93e725228"}
Sep 14 10:36:57.197351 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.197290    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m" podStartSLOduration=2.197275667 podStartE2EDuration="2.197275667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:36:57.196451821 +0000 UTC m=+705.751639748" watchObservedRunningTime="2026-09-14 10:36:57.197275667 +0000 UTC m=+705.752463592"
Sep 14 10:36:57.735043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.735000    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:57.735476 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.735450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:57.737269 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.737243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:36:57.748112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:57.748079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:58.294451 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:58.294429    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"
Sep 14 10:36:58.375288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:58.375255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c64b19-5188-4e54-9d89-4e9488a816b4-maas-api-service-ca\") pod \"c6c64b19-5188-4e54-9d89-4e9488a816b4\" (UID: \"c6c64b19-5188-4e54-9d89-4e9488a816b4\") "
Sep 14 10:36:58.375611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:58.375560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c64b19-5188-4e54-9d89-4e9488a816b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6c64b19-5188-4e54-9d89-4e9488a816b4" (UID: "c6c64b19-5188-4e54-9d89-4e9488a816b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:36:58.476260 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:58.476195    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c64b19-5188-4e54-9d89-4e9488a816b4-maas-api-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:36:58.710121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:58.710090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:59.174407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:59.174378    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"
Sep 14 10:36:59.174546 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:59.174407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn" event={"ID":"c6c64b19-5188-4e54-9d89-4e9488a816b4","Type":"ContainerDied","Data":"1512ec3a4ba88b56466ca9dc67d7db1f13c9f40c91c5c17c9eb3c09a418ca1e1"}
Sep 14 10:36:59.174546 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:59.174453    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1512ec3a4ba88b56466ca9dc67d7db1f13c9f40c91c5c17c9eb3c09a418ca1e1"
Sep 14 10:36:59.724408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:59.724377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:59.823163 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:59.823134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:36:59.947716 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:36:59.947688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:00.659414 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:00.659382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:00.873311 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:00.873279    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:37:01.443065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.443027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:01.536151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.536119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"]
Sep 14 10:37:01.536381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.536351    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm" podUID="f36e3a8d-588b-4cc4-b391-205d49b5dde2" containerName="payload-pre-processing" containerID="cri-o://aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0" gracePeriod=30
Sep 14 10:37:01.594632 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.594599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:01.690546 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.690515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:01.753828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.753791    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"]
Sep 14 10:37:01.762062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.761977    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3be523b2-bdjdn"]
Sep 14 10:37:01.790531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.790477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:01.796247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.796226    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:37:01.907628 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.907589    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36e3a8d-588b-4cc4-b391-205d49b5dde2-plugins-config-volume\") pod \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") "
Sep 14 10:37:01.908070 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.907654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t7pw\" (UniqueName: \"kubernetes.io/projected/f36e3a8d-588b-4cc4-b391-205d49b5dde2-kube-api-access-5t7pw\") pod \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\" (UID: \"f36e3a8d-588b-4cc4-b391-205d49b5dde2\") "
Sep 14 10:37:01.908070 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.908006    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36e3a8d-588b-4cc4-b391-205d49b5dde2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f36e3a8d-588b-4cc4-b391-205d49b5dde2" (UID: "f36e3a8d-588b-4cc4-b391-205d49b5dde2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:37:01.909957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.909933    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36e3a8d-588b-4cc4-b391-205d49b5dde2-kube-api-access-5t7pw" (OuterVolumeSpecName: "kube-api-access-5t7pw") pod "f36e3a8d-588b-4cc4-b391-205d49b5dde2" (UID: "f36e3a8d-588b-4cc4-b391-205d49b5dde2"). InnerVolumeSpecName "kube-api-access-5t7pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:37:01.920345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.920310    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c64b19-5188-4e54-9d89-4e9488a816b4" path="/var/lib/kubelet/pods/c6c64b19-5188-4e54-9d89-4e9488a816b4/volumes"
Sep 14 10:37:01.984492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.984418    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"]
Sep 14 10:37:01.984673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:01.984650    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy" containerID="cri-o://9fd672305faa41c7509a679ca7fcd5a79115e362cc460bcc7d82c193a1a05827" gracePeriod=30
Sep 14 10:37:02.008465 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.008431    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f36e3a8d-588b-4cc4-b391-205d49b5dde2-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:02.008465 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.008459    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t7pw\" (UniqueName: \"kubernetes.io/projected/f36e3a8d-588b-4cc4-b391-205d49b5dde2-kube-api-access-5t7pw\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:02.015668 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.015640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:02.192532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.192489    2563 generic.go:356] "Generic (PLEG): container finished" podID="f36e3a8d-588b-4cc4-b391-205d49b5dde2" containerID="aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0" exitCode=0
Sep 14 10:37:02.192720 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.192542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm" event={"ID":"f36e3a8d-588b-4cc4-b391-205d49b5dde2","Type":"ContainerDied","Data":"aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0"}
Sep 14 10:37:02.192720 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.192551    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"
Sep 14 10:37:02.192720 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.192633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm" event={"ID":"f36e3a8d-588b-4cc4-b391-205d49b5dde2","Type":"ContainerDied","Data":"13a677d871902f50a0b8fdc9ed3b435c7d83d7dda1daf8d74f068ffc77f699a6"}
Sep 14 10:37:02.192720 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.192666    2563 scope.go:117] "RemoveContainer" containerID="aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0"
Sep 14 10:37:02.205740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.205719    2563 scope.go:117] "RemoveContainer" containerID="aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0"
Sep 14 10:37:02.206209 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.206186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:02.206504 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:37:02.206442    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0\": container with ID starting with aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0 not found: ID does not exist" containerID="aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0"
Sep 14 10:37:02.206634 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.206515    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0"} err="failed to get container status \"aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0\": rpc error: code = NotFound desc = could not find container \"aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0\": container with ID starting with aceb16ddcf2b3add65f4371be46b97a9bf3a6c35809e4c42bb073fb3eacf87c0 not found: ID does not exist"
Sep 14 10:37:02.216329 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.216298    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"]
Sep 14 10:37:02.225641 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.225613    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3be523b2-6d457d8fdb-54zlm"]
Sep 14 10:37:02.477169 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.477138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:02.567842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.567792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:02.671915 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:02.671889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:03.007163 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.007123    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"]
Sep 14 10:37:03.007597 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.007557    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36e3a8d-588b-4cc4-b391-205d49b5dde2" containerName="payload-pre-processing"
Sep 14 10:37:03.007597 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.007583    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36e3a8d-588b-4cc4-b391-205d49b5dde2" containerName="payload-pre-processing"
Sep 14 10:37:03.007597 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.007595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c64b19-5188-4e54-9d89-4e9488a816b4" containerName="revoke-keys"
Sep 14 10:37:03.007698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.007600    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c64b19-5188-4e54-9d89-4e9488a816b4" containerName="revoke-keys"
Sep 14 10:37:03.007698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.007675    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36e3a8d-588b-4cc4-b391-205d49b5dde2" containerName="payload-pre-processing"
Sep 14 10:37:03.007698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.007685    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c64b19-5188-4e54-9d89-4e9488a816b4" containerName="revoke-keys"
Sep 14 10:37:03.010440 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.010422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.013160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.013139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-77bab00f-gw-openshift-default-dockercfg-9xx7r\""
Sep 14 10:37:03.045580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.045551    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"]
Sep 14 10:37:03.091779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.091748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:03.116743 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.116670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-certs\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.116743 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.116727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-token\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.116963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.116802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-data\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.116963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.116839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-credential-socket\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.116963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.116871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-envoy\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.116963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.116942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d42gc\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-kube-api-access-d42gc\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.117129 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.116987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istiod-ca-cert\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.117129 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.117061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-socket\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.117129 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.117079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-podinfo\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218300 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-socket\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218300 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-podinfo\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-certs\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-token\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-data\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-credential-socket\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-envoy\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d42gc\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-kube-api-access-d42gc\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istiod-ca-cert\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218926 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-socket\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218926 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-data\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218926 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-certs\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.218926 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.218896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-credential-socket\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.219298 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.219279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istiod-ca-cert\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.221133 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.221110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-envoy\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.221397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.221345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-podinfo\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.238607 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.238560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-token\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.238926 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.238904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d42gc\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-kube-api-access-d42gc\") pod \"e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") " pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.287232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.287152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:03.320131 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.320109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:03.421279 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.421243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:03.600492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.600311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"]
Sep 14 10:37:03.603139 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:37:03.603114    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd773afe_0d59_4e52_886c_94dbe0f4d73c.slice/crio-066d451e5234a4a416b26c47e05c9ab9e1d5df89900c5af64d03576e78334139 WatchSource:0}: Error finding container 066d451e5234a4a416b26c47e05c9ab9e1d5df89900c5af64d03576e78334139: Status 404 returned error can't find the container with id 066d451e5234a4a416b26c47e05c9ab9e1d5df89900c5af64d03576e78334139
Sep 14 10:37:03.605244 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.605214    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:03.605322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.605275    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:03.605322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.605306    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:03.919715 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:03.919682    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36e3a8d-588b-4cc4-b391-205d49b5dde2" path="/var/lib/kubelet/pods/f36e3a8d-588b-4cc4-b391-205d49b5dde2/volumes"
Sep 14 10:37:04.180851 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:04.180756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:04.207915 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:04.207879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" event={"ID":"cd773afe-0d59-4e52-886c-94dbe0f4d73c","Type":"ContainerStarted","Data":"aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9"}
Sep 14 10:37:04.208048 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:04.207954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" event={"ID":"cd773afe-0d59-4e52-886c-94dbe0f4d73c","Type":"ContainerStarted","Data":"066d451e5234a4a416b26c47e05c9ab9e1d5df89900c5af64d03576e78334139"}
Sep 14 10:37:04.235168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:04.235118    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" podStartSLOduration=2.235099601 podStartE2EDuration="2.235099601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:37:04.233125729 +0000 UTC m=+712.788313656" watchObservedRunningTime="2026-09-14 10:37:04.235099601 +0000 UTC m=+712.790287592"
Sep 14 10:37:04.320403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:04.320369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:04.325603 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:04.325564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:05.031937 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:05.031905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:05.212819 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:05.212791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:05.213680 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:05.213659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:05.709354 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:05.709324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:05.810324 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:05.810292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:06.207377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:06.207328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:06.303721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:06.303689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:37:06.304079 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:06.303762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:37:06.315751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:06.315727    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:37:06.841859 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:06.841822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:06.942679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:06.942632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:07.039001 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.038975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:07.148831 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.148799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:07.224522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.224495    2563 generic.go:356] "Generic (PLEG): container finished" podID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerID="9fd672305faa41c7509a679ca7fcd5a79115e362cc460bcc7d82c193a1a05827" exitCode=0
Sep 14 10:37:07.224661 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.224563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" event={"ID":"a96300b2-8222-4023-a4cf-07ba8c5c68f2","Type":"ContainerDied","Data":"9fd672305faa41c7509a679ca7fcd5a79115e362cc460bcc7d82c193a1a05827"}
Sep 14 10:37:07.224661 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.224652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz" event={"ID":"a96300b2-8222-4023-a4cf-07ba8c5c68f2","Type":"ContainerDied","Data":"c95ce91e640d07791ee1a86c7ccb1a0a9c9c1162e68901faf05d9ce2e676c0d2"}
Sep 14 10:37:07.224741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.224675    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c95ce91e640d07791ee1a86c7ccb1a0a9c9c1162e68901faf05d9ce2e676c0d2"
Sep 14 10:37:07.224741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.224693    2563 scope.go:117] "RemoveContainer" containerID="283b5657b735eb46a7f7cf700b4d78ace5e861ec98590e0c50df32a4e88b7dae"
Sep 14 10:37:07.231211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.231192    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:37:07.238252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.238232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:37:07.244828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.244809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:07.357730 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-data\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.357730 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357677    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-socket\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.357730 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-certs\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.357730 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357726    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-envoy\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-credential-socket\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9kfx\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-kube-api-access-s9kfx\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-token\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357909    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-podinfo\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istiod-ca-cert\") pod \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\" (UID: \"a96300b2-8222-4023-a4cf-07ba8c5c68f2\") "
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.357989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:37:07.358286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.358007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:37:07.358747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.358342    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.358747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.358350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:37:07.358747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.358362    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.358747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.358376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:37:07.358747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.358393    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.360091 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.360062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-kube-api-access-s9kfx" (OuterVolumeSpecName: "kube-api-access-s9kfx") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "kube-api-access-s9kfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:37:07.360407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.360388    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:37:07.360516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.360495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:37:07.360556 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.360498    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a96300b2-8222-4023-a4cf-07ba8c5c68f2" (UID: "a96300b2-8222-4023-a4cf-07ba8c5c68f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:37:07.459534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.459505    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9kfx\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-kube-api-access-s9kfx\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.459534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.459528    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.459534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.459538    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.459762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.459546    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.459762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.459554    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:07.459762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:07.459561    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96300b2-8222-4023-a4cf-07ba8c5c68f2-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:08.031158 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.031119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"]
Sep 14 10:37:08.031602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.031559    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy"
Sep 14 10:37:08.031602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.031598    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy"
Sep 14 10:37:08.031759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.031717    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy"
Sep 14 10:37:08.031759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.031729    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy"
Sep 14 10:37:08.036246 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.036224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.038749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.038726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-77bab00f-serving-cert\""
Sep 14 10:37:08.038889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.038756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 10:37:08.038889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.038784    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:37:08.039616 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.039594    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 10:37:08.039734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.039613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bhlx\""
Sep 14 10:37:08.049927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.049909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"]
Sep 14 10:37:08.125477 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.125427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:08.166461 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.166432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3874b8e-9160-476e-939c-154bed4d0749-openshift-service-ca\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.166612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.166468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4g8\" (UniqueName: \"kubernetes.io/projected/d3874b8e-9160-476e-939c-154bed4d0749-kube-api-access-7h4g8\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.166612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.166489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-maas-api-tls\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.166687 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.166646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-metrics-certs\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.224459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.224377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:08.236924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.236899    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"
Sep 14 10:37:08.268040 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.268003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3874b8e-9160-476e-939c-154bed4d0749-openshift-service-ca\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.268202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.268046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4g8\" (UniqueName: \"kubernetes.io/projected/d3874b8e-9160-476e-939c-154bed4d0749-kube-api-access-7h4g8\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.268202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.268071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-maas-api-tls\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.268202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.268137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-metrics-certs\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.268738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.268714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3874b8e-9160-476e-939c-154bed4d0749-openshift-service-ca\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.270619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.270597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-maas-api-tls\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.270695 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.270669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-metrics-certs\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.273216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.273196    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"]
Sep 14 10:37:08.279188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.279170    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3be523b2-openshift-default-5c76fd8768-jdnnz"]
Sep 14 10:37:08.280157 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.280133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4g8\" (UniqueName: \"kubernetes.io/projected/d3874b8e-9160-476e-939c-154bed4d0749-kube-api-access-7h4g8\") pod \"maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.308329 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.308305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:08.346399 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.346370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:08.643282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.643256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"]
Sep 14 10:37:08.645764 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:37:08.645733    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3874b8e_9160_476e_939c_154bed4d0749.slice/crio-ca636d3685a6a10339ba446c5daa829a5050eb9987343196c40638ab07d1ce53 WatchSource:0}: Error finding container ca636d3685a6a10339ba446c5daa829a5050eb9987343196c40638ab07d1ce53: Status 404 returned error can't find the container with id ca636d3685a6a10339ba446c5daa829a5050eb9987343196c40638ab07d1ce53
Sep 14 10:37:08.679976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.679944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:08.778767 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.778734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:08.881526 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.881496    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"]
Sep 14 10:37:08.881888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.881876    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy"
Sep 14 10:37:08.881936 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.881890    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" containerName="istio-proxy"
Sep 14 10:37:08.886124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.886105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:08.888512 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.888496    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-77bab00f-dockercfg-r2clb\""
Sep 14 10:37:08.888823 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.888810    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-77bab00f\""
Sep 14 10:37:08.901914 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.901865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"]
Sep 14 10:37:08.927721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.927691    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"]
Sep 14 10:37:08.931304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.931279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:08.951946 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:08.951919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"]
Sep 14 10:37:09.075399 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.075368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpw5\" (UniqueName: \"kubernetes.io/projected/2434d927-dfe1-477e-b470-9a3b2d4e92c5-kube-api-access-nqpw5\") pod \"payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") " pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:09.075565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.075431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vbv\" (UniqueName: \"kubernetes.io/projected/8c663491-4cec-4edd-98b0-815b45c31858-kube-api-access-m8vbv\") pod \"payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:09.075672 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.075559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c663491-4cec-4edd-98b0-815b45c31858-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:09.075672 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.075651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2434d927-dfe1-477e-b470-9a3b2d4e92c5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") " pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:09.176494 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.176421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vbv\" (UniqueName: \"kubernetes.io/projected/8c663491-4cec-4edd-98b0-815b45c31858-kube-api-access-m8vbv\") pod \"payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:09.176494 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.176474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c663491-4cec-4edd-98b0-815b45c31858-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:09.176761 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.176513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2434d927-dfe1-477e-b470-9a3b2d4e92c5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") " pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:09.176761 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.176536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpw5\" (UniqueName: \"kubernetes.io/projected/2434d927-dfe1-477e-b470-9a3b2d4e92c5-kube-api-access-nqpw5\") pod \"payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") " pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:09.177446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.177424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c663491-4cec-4edd-98b0-815b45c31858-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:09.177498 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.177424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2434d927-dfe1-477e-b470-9a3b2d4e92c5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") " pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:09.186949 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.186919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vbv\" (UniqueName: \"kubernetes.io/projected/8c663491-4cec-4edd-98b0-815b45c31858-kube-api-access-m8vbv\") pod \"payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:09.186949 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.186922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpw5\" (UniqueName: \"kubernetes.io/projected/2434d927-dfe1-477e-b470-9a3b2d4e92c5-kube-api-access-nqpw5\") pod \"payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") " pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:09.196756 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.196737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:09.241848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.241821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:09.247119 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.246883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth" event={"ID":"d3874b8e-9160-476e-939c-154bed4d0749","Type":"ContainerStarted","Data":"ca636d3685a6a10339ba446c5daa829a5050eb9987343196c40638ab07d1ce53"}
Sep 14 10:37:09.431186 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.431110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:09.495968 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.495867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"]
Sep 14 10:37:09.497201 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:37:09.497059    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c663491_4cec_4edd_98b0_815b45c31858.slice/crio-2f7181554a1307f016469105ad794db24b275da1c8ea1b62653685ec86623902 WatchSource:0}: Error finding container 2f7181554a1307f016469105ad794db24b275da1c8ea1b62653685ec86623902: Status 404 returned error can't find the container with id 2f7181554a1307f016469105ad794db24b275da1c8ea1b62653685ec86623902
Sep 14 10:37:09.544400 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.543851    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"]
Sep 14 10:37:09.550427 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:37:09.549354    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2434d927_dfe1_477e_b470_9a3b2d4e92c5.slice/crio-544a091eec8468b6382aba562b2b7d20f5ed3eb8fc136be1a5ce1d5eeb0bb603 WatchSource:0}: Error finding container 544a091eec8468b6382aba562b2b7d20f5ed3eb8fc136be1a5ce1d5eeb0bb603: Status 404 returned error can't find the container with id 544a091eec8468b6382aba562b2b7d20f5ed3eb8fc136be1a5ce1d5eeb0bb603
Sep 14 10:37:09.922021 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:09.921987    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96300b2-8222-4023-a4cf-07ba8c5c68f2" path="/var/lib/kubelet/pods/a96300b2-8222-4023-a4cf-07ba8c5c68f2/volumes"
Sep 14 10:37:10.256443 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.256034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt" event={"ID":"2434d927-dfe1-477e-b470-9a3b2d4e92c5","Type":"ContainerStarted","Data":"1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305"}
Sep 14 10:37:10.256443 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.256091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt" event={"ID":"2434d927-dfe1-477e-b470-9a3b2d4e92c5","Type":"ContainerStarted","Data":"544a091eec8468b6382aba562b2b7d20f5ed3eb8fc136be1a5ce1d5eeb0bb603"}
Sep 14 10:37:10.256443 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.256124    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:10.257993 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.257963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb" event={"ID":"8c663491-4cec-4edd-98b0-815b45c31858","Type":"ContainerStarted","Data":"ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098"}
Sep 14 10:37:10.258121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.258014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb" event={"ID":"8c663491-4cec-4edd-98b0-815b45c31858","Type":"ContainerStarted","Data":"2f7181554a1307f016469105ad794db24b275da1c8ea1b62653685ec86623902"}
Sep 14 10:37:10.258121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.258048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:10.276801 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.276748    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt" podStartSLOduration=2.276729041 podStartE2EDuration="2.276729041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:37:10.27313224 +0000 UTC m=+718.828320166" watchObservedRunningTime="2026-09-14 10:37:10.276729041 +0000 UTC m=+718.831916967"
Sep 14 10:37:10.297780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.297725    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb" podStartSLOduration=2.29770354 podStartE2EDuration="2.29770354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:37:10.295279895 +0000 UTC m=+718.850467823" watchObservedRunningTime="2026-09-14 10:37:10.29770354 +0000 UTC m=+718.852891472"
Sep 14 10:37:10.572679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:10.572543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:11.295348 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:11.295281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:12.177990 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:12.177948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:12.271165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:12.271128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth" event={"ID":"d3874b8e-9160-476e-939c-154bed4d0749","Type":"ContainerStarted","Data":"6d0957941eb5633a37a9f5e32295171b6543349a0e11ad7cfe520ebdcc9fe1e6"}
Sep 14 10:37:12.271330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:12.271229    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:12.294156 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:12.294111    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth" podStartSLOduration=1.6104236570000001 podStartE2EDuration="4.294098063s" totalImagesPullingTime="2.683674406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:37:12.291803285 +0000 UTC m=+720.846991209" watchObservedRunningTime="2026-09-14 10:37:12.294098063 +0000 UTC m=+720.849285996"
Sep 14 10:37:13.632677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:13.632643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:13.732820 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:13.732786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:18.281153 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:18.281125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:19.157878 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.157844    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"]
Sep 14 10:37:19.162891 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.162871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.165309 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.165290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1910e9\"/\"openshift-service-ca.crt\""
Sep 14 10:37:19.165400 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.165308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1910e9\"/\"kube-root-ca.crt\""
Sep 14 10:37:19.165764 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.165751    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1910e9\"/\"default-dockercfg-gcbgd\""
Sep 14 10:37:19.165904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.165888    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1910e9\"/\"simulator-w4-1910e9-kserve-self-signed-certs\""
Sep 14 10:37:19.172927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.172906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"]
Sep 14 10:37:19.285702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.285670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-home\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.286086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.285728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tmp-dir\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.286086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.285854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-dshm\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.286086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.285887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tls-certs\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.286086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.285918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96cb6\" (UniqueName: \"kubernetes.io/projected/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-kube-api-access-96cb6\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.286086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.285949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-model-cache\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387074 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tmp-dir\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-dshm\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tls-certs\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96cb6\" (UniqueName: \"kubernetes.io/projected/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-kube-api-access-96cb6\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-model-cache\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-home\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tmp-dir\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-home\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.387774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.387724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-model-cache\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.389455 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.389425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-dshm\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.389840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.389822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tls-certs\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.394861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.394842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96cb6\" (UniqueName: \"kubernetes.io/projected/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-kube-api-access-96cb6\") pod \"simulator-w4-1910e9-kserve-77f95785f4-9rbbl\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") " pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.472932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.472872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:19.559106 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.559073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:19.744314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:19.744291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"]
Sep 14 10:37:19.746746 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:37:19.746723    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c2a5b6e_0294_4b34_9f8c_bc6219e04467.slice/crio-d2c528f5106f81ef3309a49d3771b6cbc1cb36cd5dffc53efa1717848b970a61 WatchSource:0}: Error finding container d2c528f5106f81ef3309a49d3771b6cbc1cb36cd5dffc53efa1717848b970a61: Status 404 returned error can't find the container with id d2c528f5106f81ef3309a49d3771b6cbc1cb36cd5dffc53efa1717848b970a61
Sep 14 10:37:20.314596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:20.314529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl" event={"ID":"8c2a5b6e-0294-4b34-9f8c-bc6219e04467","Type":"ContainerStarted","Data":"83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e"}
Sep 14 10:37:20.314596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:20.314603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl" event={"ID":"8c2a5b6e-0294-4b34-9f8c-bc6219e04467","Type":"ContainerStarted","Data":"d2c528f5106f81ef3309a49d3771b6cbc1cb36cd5dffc53efa1717848b970a61"}
Sep 14 10:37:20.330628 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:20.330579    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl" podStartSLOduration=1.33055135 podStartE2EDuration="1.33055135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:37:20.330007494 +0000 UTC m=+728.885195417" watchObservedRunningTime="2026-09-14 10:37:20.33055135 +0000 UTC m=+728.885739275"
Sep 14 10:37:20.667183 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:20.667148    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:37:21.266987 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:21.266957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:21.267168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:21.267153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:26.892517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:26.892481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:29.473336 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:29.473298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:29.473744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:29.473363    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:29.486009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:29.485988    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:30.382780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:30.382749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:30.384965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:30.384943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:37:31.318663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:31.318631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:31.379944 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:31.379911    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerID="aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9" exitCode=0
Sep 14 10:37:31.380108 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:31.379980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" event={"ID":"cd773afe-0d59-4e52-886c-94dbe0f4d73c","Type":"ContainerDied","Data":"aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9"}
Sep 14 10:37:31.380486 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:31.380468    2563 scope.go:117] "RemoveContainer" containerID="aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9"
Sep 14 10:37:31.381562 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:31.381537    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:31.381649 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:31.381634    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:31.381685 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:31.381668    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:32.210097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:32.210061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:32.387201 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:32.387158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" event={"ID":"cd773afe-0d59-4e52-886c-94dbe0f4d73c","Type":"ContainerStarted","Data":"ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513"}
Sep 14 10:37:32.920662 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:32.920609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:33.320388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:33.320249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:33.321855 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:33.321810    2563 patch_prober.go:28] interesting pod/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 14 10:37:33.322085 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:33.322058    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 14 10:37:33.789887 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:33.789853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:34.321304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:34.321258    2563 patch_prober.go:28] interesting pod/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 14 10:37:34.321511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:34.321328    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 14 10:37:34.471477 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:34.471438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:35.111728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:35.111689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:35.213143 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:35.213109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:35.320838 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:35.320806    2563 patch_prober.go:28] interesting pod/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 14 10:37:35.321115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:35.321085    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 14 10:37:35.566239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:35.566205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:37:36.092494 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:36.092460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:36.321518 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:36.321482    2563 patch_prober.go:28] interesting pod/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 14 10:37:36.322087 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:36.322054    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 14 10:37:37.324238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:37.324209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:37.325136 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:37.325116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:37:56.852203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:56.852168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:57.386202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.386168    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"]
Sep 14 10:37:57.386446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.386406    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth" podUID="d3874b8e-9160-476e-939c-154bed4d0749" containerName="maas-api" containerID="cri-o://6d0957941eb5633a37a9f5e32295171b6543349a0e11ad7cfe520ebdcc9fe1e6" gracePeriod=30
Sep 14 10:37:57.416926 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.416896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:57.449800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.449768    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"]
Sep 14 10:37:57.450033 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.449996    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt" podUID="2434d927-dfe1-477e-b470-9a3b2d4e92c5" containerName="payload-processing" containerID="cri-o://1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305" gracePeriod=30
Sep 14 10:37:57.456517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.456483    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"]
Sep 14 10:37:57.456770 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.456743    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb" podUID="8c663491-4cec-4edd-98b0-815b45c31858" containerName="payload-pre-processing" containerID="cri-o://ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098" gracePeriod=30
Sep 14 10:37:57.493709 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.493676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:57.524022 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.523992    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3874b8e-9160-476e-939c-154bed4d0749" containerID="6d0957941eb5633a37a9f5e32295171b6543349a0e11ad7cfe520ebdcc9fe1e6" exitCode=0
Sep 14 10:37:57.524149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.524034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth" event={"ID":"d3874b8e-9160-476e-939c-154bed4d0749","Type":"ContainerDied","Data":"6d0957941eb5633a37a9f5e32295171b6543349a0e11ad7cfe520ebdcc9fe1e6"}
Sep 14 10:37:57.560509 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.560470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:57.622446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.622416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:57.682406 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.681556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:57.708199 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.708172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:57.734731 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.734700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:57.744355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.744323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:57.754175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.754127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-metrics-certs\") pod \"d3874b8e-9160-476e-939c-154bed4d0749\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") "
Sep 14 10:37:57.754175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.754163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h4g8\" (UniqueName: \"kubernetes.io/projected/d3874b8e-9160-476e-939c-154bed4d0749-kube-api-access-7h4g8\") pod \"d3874b8e-9160-476e-939c-154bed4d0749\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") "
Sep 14 10:37:57.754512 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.754210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3874b8e-9160-476e-939c-154bed4d0749-openshift-service-ca\") pod \"d3874b8e-9160-476e-939c-154bed4d0749\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") "
Sep 14 10:37:57.754512 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.754226    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-maas-api-tls\") pod \"d3874b8e-9160-476e-939c-154bed4d0749\" (UID: \"d3874b8e-9160-476e-939c-154bed4d0749\") "
Sep 14 10:37:57.755459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.754899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"]
Sep 14 10:37:57.755459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.755130    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy" containerID="cri-o://ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513" gracePeriod=30
Sep 14 10:37:57.755459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.755131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3874b8e-9160-476e-939c-154bed4d0749-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3874b8e-9160-476e-939c-154bed4d0749" (UID: "d3874b8e-9160-476e-939c-154bed4d0749"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:37:57.760750 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.759049    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3874b8e-9160-476e-939c-154bed4d0749" (UID: "d3874b8e-9160-476e-939c-154bed4d0749"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:37:57.760750 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.760382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3874b8e-9160-476e-939c-154bed4d0749" (UID: "d3874b8e-9160-476e-939c-154bed4d0749"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:37:57.763952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.763920    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3874b8e-9160-476e-939c-154bed4d0749-kube-api-access-7h4g8" (OuterVolumeSpecName: "kube-api-access-7h4g8") pod "d3874b8e-9160-476e-939c-154bed4d0749" (UID: "d3874b8e-9160-476e-939c-154bed4d0749"). InnerVolumeSpecName "kube-api-access-7h4g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:37:57.784340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.784300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:57.855918 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.855875    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2434d927-dfe1-477e-b470-9a3b2d4e92c5-plugins-config-volume\") pod \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") "
Sep 14 10:37:57.856910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.856886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c663491-4cec-4edd-98b0-815b45c31858-plugins-config-volume\") pod \"8c663491-4cec-4edd-98b0-815b45c31858\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") "
Sep 14 10:37:57.857519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.857499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8vbv\" (UniqueName: \"kubernetes.io/projected/8c663491-4cec-4edd-98b0-815b45c31858-kube-api-access-m8vbv\") pod \"8c663491-4cec-4edd-98b0-815b45c31858\" (UID: \"8c663491-4cec-4edd-98b0-815b45c31858\") "
Sep 14 10:37:57.857730 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.857716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqpw5\" (UniqueName: \"kubernetes.io/projected/2434d927-dfe1-477e-b470-9a3b2d4e92c5-kube-api-access-nqpw5\") pod \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\" (UID: \"2434d927-dfe1-477e-b470-9a3b2d4e92c5\") "
Sep 14 10:37:57.858147 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.858130    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:57.858249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.858236    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h4g8\" (UniqueName: \"kubernetes.io/projected/d3874b8e-9160-476e-939c-154bed4d0749-kube-api-access-7h4g8\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:57.858330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.858318    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3874b8e-9160-476e-939c-154bed4d0749-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:57.858404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.858393    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3874b8e-9160-476e-939c-154bed4d0749-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:57.858939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.856769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2434d927-dfe1-477e-b470-9a3b2d4e92c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2434d927-dfe1-477e-b470-9a3b2d4e92c5" (UID: "2434d927-dfe1-477e-b470-9a3b2d4e92c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:37:57.858939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.857421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c663491-4cec-4edd-98b0-815b45c31858-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c663491-4cec-4edd-98b0-815b45c31858" (UID: "8c663491-4cec-4edd-98b0-815b45c31858"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:37:57.868924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.868899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2434d927-dfe1-477e-b470-9a3b2d4e92c5-kube-api-access-nqpw5" (OuterVolumeSpecName: "kube-api-access-nqpw5") pod "2434d927-dfe1-477e-b470-9a3b2d4e92c5" (UID: "2434d927-dfe1-477e-b470-9a3b2d4e92c5"). InnerVolumeSpecName "kube-api-access-nqpw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:37:57.880594 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.879704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c663491-4cec-4edd-98b0-815b45c31858-kube-api-access-m8vbv" (OuterVolumeSpecName: "kube-api-access-m8vbv") pod "8c663491-4cec-4edd-98b0-815b45c31858" (UID: "8c663491-4cec-4edd-98b0-815b45c31858"). InnerVolumeSpecName "kube-api-access-m8vbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:37:57.959157 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.959128    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpw5\" (UniqueName: \"kubernetes.io/projected/2434d927-dfe1-477e-b470-9a3b2d4e92c5-kube-api-access-nqpw5\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:57.959157 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.959153    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2434d927-dfe1-477e-b470-9a3b2d4e92c5-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:57.959287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.959162    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c663491-4cec-4edd-98b0-815b45c31858-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:57.959287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:57.959171    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vbv\" (UniqueName: \"kubernetes.io/projected/8c663491-4cec-4edd-98b0-815b45c31858-kube-api-access-m8vbv\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:37:58.016164 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.016140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:58.233482 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.233400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:58.515749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.515660    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"]
Sep 14 10:37:58.516108 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516093    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2434d927-dfe1-477e-b470-9a3b2d4e92c5" containerName="payload-processing"
Sep 14 10:37:58.516108 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2434d927-dfe1-477e-b470-9a3b2d4e92c5" containerName="payload-processing"
Sep 14 10:37:58.516232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516140    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3874b8e-9160-476e-939c-154bed4d0749" containerName="maas-api"
Sep 14 10:37:58.516232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516148    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3874b8e-9160-476e-939c-154bed4d0749" containerName="maas-api"
Sep 14 10:37:58.516232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516157    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c663491-4cec-4edd-98b0-815b45c31858" containerName="payload-pre-processing"
Sep 14 10:37:58.516232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516162    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c663491-4cec-4edd-98b0-815b45c31858" containerName="payload-pre-processing"
Sep 14 10:37:58.516232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516218    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2434d927-dfe1-477e-b470-9a3b2d4e92c5" containerName="payload-processing"
Sep 14 10:37:58.516232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516230    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3874b8e-9160-476e-939c-154bed4d0749" containerName="maas-api"
Sep 14 10:37:58.516507 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.516244    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c663491-4cec-4edd-98b0-815b45c31858" containerName="payload-pre-processing"
Sep 14 10:37:58.519608 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.519580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.521775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.521758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-18c04da2-openshift-default-dockercfg-t89q9\""
Sep 14 10:37:58.532755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.532727    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c663491-4cec-4edd-98b0-815b45c31858" containerID="ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098" exitCode=0
Sep 14 10:37:58.532854 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.532779    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"
Sep 14 10:37:58.532854 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.532812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb" event={"ID":"8c663491-4cec-4edd-98b0-815b45c31858","Type":"ContainerDied","Data":"ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098"}
Sep 14 10:37:58.532970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.532868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb" event={"ID":"8c663491-4cec-4edd-98b0-815b45c31858","Type":"ContainerDied","Data":"2f7181554a1307f016469105ad794db24b275da1c8ea1b62653685ec86623902"}
Sep 14 10:37:58.532970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.532899    2563 scope.go:117] "RemoveContainer" containerID="ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098"
Sep 14 10:37:58.533069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.532961    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"]
Sep 14 10:37:58.535231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.535215    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"
Sep 14 10:37:58.535465 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.535440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth" event={"ID":"d3874b8e-9160-476e-939c-154bed4d0749","Type":"ContainerDied","Data":"ca636d3685a6a10339ba446c5daa829a5050eb9987343196c40638ab07d1ce53"}
Sep 14 10:37:58.537876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.537852    2563 generic.go:356] "Generic (PLEG): container finished" podID="2434d927-dfe1-477e-b470-9a3b2d4e92c5" containerID="1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305" exitCode=0
Sep 14 10:37:58.537991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.537934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt" event={"ID":"2434d927-dfe1-477e-b470-9a3b2d4e92c5","Type":"ContainerDied","Data":"1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305"}
Sep 14 10:37:58.537991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.537951    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"
Sep 14 10:37:58.537991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.537969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt" event={"ID":"2434d927-dfe1-477e-b470-9a3b2d4e92c5","Type":"ContainerDied","Data":"544a091eec8468b6382aba562b2b7d20f5ed3eb8fc136be1a5ce1d5eeb0bb603"}
Sep 14 10:37:58.543084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.543065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:58.544253 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.544238    2563 scope.go:117] "RemoveContainer" containerID="ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098"
Sep 14 10:37:58.544553 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:37:58.544533    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098\": container with ID starting with ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098 not found: ID does not exist" containerID="ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098"
Sep 14 10:37:58.544702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.544560    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098"} err="failed to get container status \"ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098\": rpc error: code = NotFound desc = could not find container \"ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098\": container with ID starting with ce333f0e8e51b7851ed7163b14a48f6943e54b4f2fda4df38abc88b8099b0098 not found: ID does not exist"
Sep 14 10:37:58.544702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.544593    2563 scope.go:117] "RemoveContainer" containerID="6d0957941eb5633a37a9f5e32295171b6543349a0e11ad7cfe520ebdcc9fe1e6"
Sep 14 10:37:58.556178 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.556157    2563 scope.go:117] "RemoveContainer" containerID="1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305"
Sep 14 10:37:58.564988 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.564960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-certs\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565113 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d388bf6b-072a-46a8-8066-7f18368bb919-istiod-ca-cert\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565113 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-envoy\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565113 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-socket\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-credential-socket\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2578v\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-kube-api-access-2578v\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565234    2563 scope.go:117] "RemoveContainer" containerID="1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305"
Sep 14 10:37:58.565402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d388bf6b-072a-46a8-8066-7f18368bb919-istio-podinfo\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-istio-token\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-data\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.565686 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:37:58.565536    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305\": container with ID starting with 1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305 not found: ID does not exist" containerID="1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305"
Sep 14 10:37:58.565686 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.565608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305"} err="failed to get container status \"1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305\": rpc error: code = NotFound desc = could not find container \"1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305\": container with ID starting with 1d3d367c1a1111aceb9d91c9f200a2831f059f4430558cea4e973939322e1305 not found: ID does not exist"
Sep 14 10:37:58.566216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.566195    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"]
Sep 14 10:37:58.571412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.571376    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-77bab00f-d5f869458-jt8rt"]
Sep 14 10:37:58.581021 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.580999    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"]
Sep 14 10:37:58.586196 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.586175    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-77bab00f-d978bd5dd-ffbth"]
Sep 14 10:37:58.595148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.595125    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"]
Sep 14 10:37:58.596984 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.596965    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-77bab00f-65685fbb8b-8j4zb"]
Sep 14 10:37:58.666197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-istio-token\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-data\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-certs\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d388bf6b-072a-46a8-8066-7f18368bb919-istiod-ca-cert\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-envoy\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-socket\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-credential-socket\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2578v\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-kube-api-access-2578v\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d388bf6b-072a-46a8-8066-7f18368bb919-istio-podinfo\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-socket\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.666870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.666840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-data\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.667075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.667051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-certs\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.667075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.667069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d388bf6b-072a-46a8-8066-7f18368bb919-istiod-ca-cert\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.667215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.667060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-credential-socket\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.668681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.668659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-envoy\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.668913 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.668896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d388bf6b-072a-46a8-8066-7f18368bb919-istio-podinfo\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.674436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.674408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-istio-token\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.674667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.674650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2578v\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-kube-api-access-2578v\") pod \"e2e-ait-18c04da2-openshift-default-555c79855d-vth4r\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") " pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.684504 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.684480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:58.830008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.829930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:58.968820 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:58.968785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:59.104309 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.104280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"]
Sep 14 10:37:59.106067 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:37:59.106044    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd388bf6b_072a_46a8_8066_7f18368bb919.slice/crio-f4bd6e0de9d3d89b80feb593a6e91e8a3cc99da429ccb7c043468e58de7bc040 WatchSource:0}: Error finding container f4bd6e0de9d3d89b80feb593a6e91e8a3cc99da429ccb7c043468e58de7bc040: Status 404 returned error can't find the container with id f4bd6e0de9d3d89b80feb593a6e91e8a3cc99da429ccb7c043468e58de7bc040
Sep 14 10:37:59.108192 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.108165    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:59.108261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.108223    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:59.108261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.108250    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:37:59.546337 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.546298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" event={"ID":"d388bf6b-072a-46a8-8066-7f18368bb919","Type":"ContainerStarted","Data":"dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe"}
Sep 14 10:37:59.546537 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.546355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" event={"ID":"d388bf6b-072a-46a8-8066-7f18368bb919","Type":"ContainerStarted","Data":"f4bd6e0de9d3d89b80feb593a6e91e8a3cc99da429ccb7c043468e58de7bc040"}
Sep 14 10:37:59.568427 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.568370    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" podStartSLOduration=1.5683541779999999 podStartE2EDuration="1.568354178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:37:59.565969847 +0000 UTC m=+768.121157772" watchObservedRunningTime="2026-09-14 10:37:59.568354178 +0000 UTC m=+768.123542169"
Sep 14 10:37:59.692462 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.692432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:37:59.830779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.830705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:37:59.919034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.919006    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2434d927-dfe1-477e-b470-9a3b2d4e92c5" path="/var/lib/kubelet/pods/2434d927-dfe1-477e-b470-9a3b2d4e92c5/volumes"
Sep 14 10:37:59.919359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.919346    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c663491-4cec-4edd-98b0-815b45c31858" path="/var/lib/kubelet/pods/8c663491-4cec-4edd-98b0-815b45c31858/volumes"
Sep 14 10:37:59.919702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:37:59.919689    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3874b8e-9160-476e-939c-154bed4d0749" path="/var/lib/kubelet/pods/d3874b8e-9160-476e-939c-154bed4d0749/volumes"
Sep 14 10:38:00.503519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:00.503429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:00.835997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:00.835920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:01.125516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:01.125454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:01.213771 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:01.213702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:01.318759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:01.318721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:01.558916 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:01.558824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:01.559917 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:01.559894    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:02.095800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.095765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:02.906913 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.906878    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"]
Sep 14 10:38:02.910231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.910208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:02.912428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.912408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:38:02.912685 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.912670    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 10:38:02.913030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.913014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-18c04da2-serving-cert\""
Sep 14 10:38:02.913173 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.913050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bhlx\""
Sep 14 10:38:02.913173 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.913062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 10:38:02.923980 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.923955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"]
Sep 14 10:38:02.961726 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:02.961684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:03.009459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.009423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae65f7-e54f-487f-9ce6-65a47c117d64-openshift-service-ca\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.009673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.009469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-metrics-certs\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.009673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.009640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl62c\" (UniqueName: \"kubernetes.io/projected/85ae65f7-e54f-487f-9ce6-65a47c117d64-kube-api-access-rl62c\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.009673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.009666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-maas-api-tls\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.061946 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.061916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:03.107865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.107838    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:38:03.110992 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.110969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-metrics-certs\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.111119 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.111050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl62c\" (UniqueName: \"kubernetes.io/projected/85ae65f7-e54f-487f-9ce6-65a47c117d64-kube-api-access-rl62c\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.111119 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.111069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-maas-api-tls\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.111231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.111145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae65f7-e54f-487f-9ce6-65a47c117d64-openshift-service-ca\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.111748 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.111727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae65f7-e54f-487f-9ce6-65a47c117d64-openshift-service-ca\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.113410 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.113384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-metrics-certs\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.113499 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.113444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-maas-api-tls\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.119222 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.119203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl62c\" (UniqueName: \"kubernetes.io/projected/85ae65f7-e54f-487f-9ce6-65a47c117d64-kube-api-access-rl62c\") pod \"maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.132610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.132583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:03.211956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.211894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-podinfo\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.211956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.211936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-data\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212131 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.211977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-envoy\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212131 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d42gc\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-kube-api-access-d42gc\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212131 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-certs\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212131 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-token\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212149    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-credential-socket\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212186    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istiod-ca-cert\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:03.212466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212331    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:03.212466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-socket\") pod \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\" (UID: \"cd773afe-0d59-4e52-886c-94dbe0f4d73c\") "
Sep 14 10:38:03.212606 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212509    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:38:03.212606 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:03.212815 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212798    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.212874 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212822    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.212874 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212868    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.212953 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212882    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.212953 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.212907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:03.214147 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.214129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:03.214246 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.214229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-kube-api-access-d42gc" (OuterVolumeSpecName: "kube-api-access-d42gc") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "kube-api-access-d42gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:38:03.214643 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.214626    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:38:03.214788 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.214774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd773afe-0d59-4e52-886c-94dbe0f4d73c" (UID: "cd773afe-0d59-4e52-886c-94dbe0f4d73c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:38:03.268075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.268052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:03.314044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.313903    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d42gc\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-kube-api-access-d42gc\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.314044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.313934    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.314044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.313947    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.314044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.313959    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.314044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.313971    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd773afe-0d59-4e52-886c-94dbe0f4d73c-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:03.465176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.465082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:03.538320 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.538293    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"]
Sep 14 10:38:03.540717 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:38:03.540684    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ae65f7_e54f_487f_9ce6_65a47c117d64.slice/crio-e6b7531dd0534656debc83e7372766984007c2ab57f1dd9f37ae52a9f2936e9f WatchSource:0}: Error finding container e6b7531dd0534656debc83e7372766984007c2ab57f1dd9f37ae52a9f2936e9f: Status 404 returned error can't find the container with id e6b7531dd0534656debc83e7372766984007c2ab57f1dd9f37ae52a9f2936e9f
Sep 14 10:38:03.541290 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.541267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:03.575504 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.574740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz" event={"ID":"85ae65f7-e54f-487f-9ce6-65a47c117d64","Type":"ContainerStarted","Data":"e6b7531dd0534656debc83e7372766984007c2ab57f1dd9f37ae52a9f2936e9f"}
Sep 14 10:38:03.577751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.577710    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerID="ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513" exitCode=0
Sep 14 10:38:03.577894 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.577854    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"
Sep 14 10:38:03.577894 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.577867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" event={"ID":"cd773afe-0d59-4e52-886c-94dbe0f4d73c","Type":"ContainerDied","Data":"ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513"}
Sep 14 10:38:03.578003 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.577920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884" event={"ID":"cd773afe-0d59-4e52-886c-94dbe0f4d73c","Type":"ContainerDied","Data":"066d451e5234a4a416b26c47e05c9ab9e1d5df89900c5af64d03576e78334139"}
Sep 14 10:38:03.578003 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.577944    2563 scope.go:117] "RemoveContainer" containerID="ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513"
Sep 14 10:38:03.579482 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.579453    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"]
Sep 14 10:38:03.579899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.579883    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy"
Sep 14 10:38:03.579899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.579900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy"
Sep 14 10:38:03.580011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.579986    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy"
Sep 14 10:38:03.580011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.579995    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy"
Sep 14 10:38:03.583094 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.583076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.586289 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.586267    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-18c04da2\""
Sep 14 10:38:03.586562 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.586522    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-18c04da2-dockercfg-rxb4z\""
Sep 14 10:38:03.589009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.588993    2563 scope.go:117] "RemoveContainer" containerID="aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9"
Sep 14 10:38:03.589190 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.589171    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"]
Sep 14 10:38:03.589652 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.589632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy"
Sep 14 10:38:03.589652 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.589651    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" containerName="istio-proxy"
Sep 14 10:38:03.592780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.592763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.597800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.597782    2563 scope.go:117] "RemoveContainer" containerID="ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513"
Sep 14 10:38:03.598080 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:38:03.598058    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513\": container with ID starting with ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513 not found: ID does not exist" containerID="ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513"
Sep 14 10:38:03.598148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.598086    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513"} err="failed to get container status \"ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513\": rpc error: code = NotFound desc = could not find container \"ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513\": container with ID starting with ea7b045c5c9bd7d690d6a2fd494230fb6f43d4b2946c3fa16ca8aad90f6f9513 not found: ID does not exist"
Sep 14 10:38:03.598148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.598108    2563 scope.go:117] "RemoveContainer" containerID="aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9"
Sep 14 10:38:03.598412 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:38:03.598395    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9\": container with ID starting with aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9 not found: ID does not exist" containerID="aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9"
Sep 14 10:38:03.598467 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.598421    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9"} err="failed to get container status \"aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9\": rpc error: code = NotFound desc = could not find container \"aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9\": container with ID starting with aa28b2c1b07bb5a986ec74322223e07913ca86bf742ce0425924b5859e6e12f9 not found: ID does not exist"
Sep 14 10:38:03.606631 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.606608    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"]
Sep 14 10:38:03.608796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.608773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"]
Sep 14 10:38:03.632944 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.632918    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"]
Sep 14 10:38:03.642655 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.642599    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-77bab00f-gw-openshift-default-845565d4b4-ls884"]
Sep 14 10:38:03.717718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.717616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7f4345-b132-4b1b-a006-9e81755f362e-plugins-config-volume\") pod \"payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") " pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.717718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.717670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/dc9251f4-0d28-479d-a2e1-a4602ad42435-kube-api-access-j9m88\") pod \"payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.717718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.717717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgthb\" (UniqueName: \"kubernetes.io/projected/1a7f4345-b132-4b1b-a006-9e81755f362e-kube-api-access-qgthb\") pod \"payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") " pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.718009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.717917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9251f4-0d28-479d-a2e1-a4602ad42435-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.818800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.818766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/dc9251f4-0d28-479d-a2e1-a4602ad42435-kube-api-access-j9m88\") pod \"payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.818800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.818802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgthb\" (UniqueName: \"kubernetes.io/projected/1a7f4345-b132-4b1b-a006-9e81755f362e-kube-api-access-qgthb\") pod \"payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") " pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.818976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.818850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9251f4-0d28-479d-a2e1-a4602ad42435-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.818976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.818914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7f4345-b132-4b1b-a006-9e81755f362e-plugins-config-volume\") pod \"payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") " pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.819550 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.819531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7f4345-b132-4b1b-a006-9e81755f362e-plugins-config-volume\") pod \"payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") " pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.819610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.819547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9251f4-0d28-479d-a2e1-a4602ad42435-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.829271 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.829238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgthb\" (UniqueName: \"kubernetes.io/projected/1a7f4345-b132-4b1b-a006-9e81755f362e-kube-api-access-qgthb\") pod \"payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") " pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.829402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.829297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/dc9251f4-0d28-479d-a2e1-a4602ad42435-kube-api-access-j9m88\") pod \"payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.896963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.896936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:03.906857 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.906187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:03.920786 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:03.920763    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd773afe-0d59-4e52-886c-94dbe0f4d73c" path="/var/lib/kubelet/pods/cd773afe-0d59-4e52-886c-94dbe0f4d73c/volumes"
Sep 14 10:38:04.014534 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.014503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:04.174347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.174323    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"]
Sep 14 10:38:04.176521 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:38:04.176494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9251f4_0d28_479d_a2e1_a4602ad42435.slice/crio-9c0367ab45bd43928354464fe2be3da9fe2cc46004585943200a1888ae9114a9 WatchSource:0}: Error finding container 9c0367ab45bd43928354464fe2be3da9fe2cc46004585943200a1888ae9114a9: Status 404 returned error can't find the container with id 9c0367ab45bd43928354464fe2be3da9fe2cc46004585943200a1888ae9114a9
Sep 14 10:38:04.187135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.187115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"]
Sep 14 10:38:04.190442 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:38:04.190415    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7f4345_b132_4b1b_a006_9e81755f362e.slice/crio-5c06daafb9944000f75c4626913794e8646fa384246b227cd3f71629aaee6f90 WatchSource:0}: Error finding container 5c06daafb9944000f75c4626913794e8646fa384246b227cd3f71629aaee6f90: Status 404 returned error can't find the container with id 5c06daafb9944000f75c4626913794e8646fa384246b227cd3f71629aaee6f90
Sep 14 10:38:04.586089 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.585978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8" event={"ID":"dc9251f4-0d28-479d-a2e1-a4602ad42435","Type":"ContainerStarted","Data":"dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654"}
Sep 14 10:38:04.586089 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.586047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:04.586089 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.586071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8" event={"ID":"dc9251f4-0d28-479d-a2e1-a4602ad42435","Type":"ContainerStarted","Data":"9c0367ab45bd43928354464fe2be3da9fe2cc46004585943200a1888ae9114a9"}
Sep 14 10:38:04.587642 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.587606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz" event={"ID":"85ae65f7-e54f-487f-9ce6-65a47c117d64","Type":"ContainerStarted","Data":"366f55d70414eac8cf382749da9b5fdd0594327a6904e1d336f6552de541c716"}
Sep 14 10:38:04.587798 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.587701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:04.590058 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.590036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg" event={"ID":"1a7f4345-b132-4b1b-a006-9e81755f362e","Type":"ContainerStarted","Data":"778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b"}
Sep 14 10:38:04.590193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.590078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg" event={"ID":"1a7f4345-b132-4b1b-a006-9e81755f362e","Type":"ContainerStarted","Data":"5c06daafb9944000f75c4626913794e8646fa384246b227cd3f71629aaee6f90"}
Sep 14 10:38:04.590193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.590117    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:04.603776 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.603730    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8" podStartSLOduration=1.603714207 podStartE2EDuration="1.603714207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:38:04.602717647 +0000 UTC m=+773.157905572" watchObservedRunningTime="2026-09-14 10:38:04.603714207 +0000 UTC m=+773.158902133"
Sep 14 10:38:04.622413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.622365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg" podStartSLOduration=1.622347816 podStartE2EDuration="1.622347816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:38:04.61995377 +0000 UTC m=+773.175141706" watchObservedRunningTime="2026-09-14 10:38:04.622347816 +0000 UTC m=+773.177535742"
Sep 14 10:38:04.640117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.640050    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz" podStartSLOduration=2.293233642 podStartE2EDuration="2.640030528s" totalImagesPullingTime="346.796886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:38:04.637503814 +0000 UTC m=+773.192691740" watchObservedRunningTime="2026-09-14 10:38:04.640030528 +0000 UTC m=+773.195218455"
Sep 14 10:38:04.790805 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:04.790770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:05.570834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:05.570800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:38:05.599130 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:05.599097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:06.264277 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:06.264242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:10.599473 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:10.599447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:15.597266 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:15.597223    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:15.597660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:15.597454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:24.061326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:24.061284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:38:25.089670 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:25.089636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:25.780521 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:25.780486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:26.370510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.370478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:26.607957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.607921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:26.708098 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.708069    2563 generic.go:356] "Generic (PLEG): container finished" podID="d388bf6b-072a-46a8-8066-7f18368bb919" containerID="dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe" exitCode=0
Sep 14 10:38:26.708256 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.708141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" event={"ID":"d388bf6b-072a-46a8-8066-7f18368bb919","Type":"ContainerDied","Data":"dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe"}
Sep 14 10:38:26.708556 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.708542    2563 scope.go:117] "RemoveContainer" containerID="dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe"
Sep 14 10:38:26.709706 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.709682    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:26.709768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.709744    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:26.709803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:26.709775    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:27.497388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:27.497345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:27.719665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:27.719623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" event={"ID":"d388bf6b-072a-46a8-8066-7f18368bb919","Type":"ContainerStarted","Data":"ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff"}
Sep 14 10:38:27.831257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:27.831159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:27.832223 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:27.832193    2563 patch_prober.go:28] interesting pod/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 10:38:27.832445 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:27.832420    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 10:38:28.367446 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:28.367404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:28.830948 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:28.830912    2563 patch_prober.go:28] interesting pod/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 10:38:28.831524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:28.831490    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 10:38:29.011829 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:29.011786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:29.695862 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:29.695829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:29.831209 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:29.831171    2563 patch_prober.go:28] interesting pod/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 10:38:29.831804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:29.831770    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 10:38:30.501256 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:30.501220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:30.830901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:30.830798    2563 patch_prober.go:28] interesting pod/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 10:38:30.830901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:30.830886    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 10:38:31.190290 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:31.190255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:31.559259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:31.559176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:31.830819 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:31.830729    2563 patch_prober.go:28] interesting pod/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 14 10:38:31.831096 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:31.831058    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 14 10:38:31.901085 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:31.901054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:32.609072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.609033    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"]
Sep 14 10:38:32.609831 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.609798    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz" podUID="85ae65f7-e54f-487f-9ce6-65a47c117d64" containerName="maas-api" containerID="cri-o://366f55d70414eac8cf382749da9b5fdd0594327a6904e1d336f6552de541c716" gracePeriod=30
Sep 14 10:38:32.640310 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.640279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:32.665051 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.665011    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"]
Sep 14 10:38:32.665267 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.665239    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg" podUID="1a7f4345-b132-4b1b-a006-9e81755f362e" containerName="payload-processing" containerID="cri-o://778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b" gracePeriod=30
Sep 14 10:38:32.674635 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.674602    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"]
Sep 14 10:38:32.675067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.675033    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8" podUID="dc9251f4-0d28-479d-a2e1-a4602ad42435" containerName="payload-pre-processing" containerID="cri-o://dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654" gracePeriod=30
Sep 14 10:38:32.741458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.741426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:32.756470 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.756439    2563 generic.go:356] "Generic (PLEG): container finished" podID="85ae65f7-e54f-487f-9ce6-65a47c117d64" containerID="366f55d70414eac8cf382749da9b5fdd0594327a6904e1d336f6552de541c716" exitCode=0
Sep 14 10:38:32.756638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.756505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz" event={"ID":"85ae65f7-e54f-487f-9ce6-65a47c117d64","Type":"ContainerDied","Data":"366f55d70414eac8cf382749da9b5fdd0594327a6904e1d336f6552de541c716"}
Sep 14 10:38:32.836119 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.836034    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:32.838775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.838718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:32.843972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.843918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:32.943168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.943119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:32.952216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.952182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:32.955752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.955726    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:32.975594 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.975539    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:32.989877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.989849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl62c\" (UniqueName: \"kubernetes.io/projected/85ae65f7-e54f-487f-9ce6-65a47c117d64-kube-api-access-rl62c\") pod \"85ae65f7-e54f-487f-9ce6-65a47c117d64\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") "
Sep 14 10:38:32.990011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.989927    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-metrics-certs\") pod \"85ae65f7-e54f-487f-9ce6-65a47c117d64\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") "
Sep 14 10:38:32.990011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.989960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7f4345-b132-4b1b-a006-9e81755f362e-plugins-config-volume\") pod \"1a7f4345-b132-4b1b-a006-9e81755f362e\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") "
Sep 14 10:38:32.990107 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.990033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae65f7-e54f-487f-9ce6-65a47c117d64-openshift-service-ca\") pod \"85ae65f7-e54f-487f-9ce6-65a47c117d64\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") "
Sep 14 10:38:32.990107 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.990081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-maas-api-tls\") pod \"85ae65f7-e54f-487f-9ce6-65a47c117d64\" (UID: \"85ae65f7-e54f-487f-9ce6-65a47c117d64\") "
Sep 14 10:38:32.990216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.990105    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgthb\" (UniqueName: \"kubernetes.io/projected/1a7f4345-b132-4b1b-a006-9e81755f362e-kube-api-access-qgthb\") pod \"1a7f4345-b132-4b1b-a006-9e81755f362e\" (UID: \"1a7f4345-b132-4b1b-a006-9e81755f362e\") "
Sep 14 10:38:32.990408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.990371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7f4345-b132-4b1b-a006-9e81755f362e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a7f4345-b132-4b1b-a006-9e81755f362e" (UID: "1a7f4345-b132-4b1b-a006-9e81755f362e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:38:32.990668 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.990644    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7f4345-b132-4b1b-a006-9e81755f362e-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:32.990914 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.990830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ae65f7-e54f-487f-9ce6-65a47c117d64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85ae65f7-e54f-487f-9ce6-65a47c117d64" (UID: "85ae65f7-e54f-487f-9ce6-65a47c117d64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:38:32.992215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.992105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "85ae65f7-e54f-487f-9ce6-65a47c117d64" (UID: "85ae65f7-e54f-487f-9ce6-65a47c117d64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:38:32.992215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.992187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ae65f7-e54f-487f-9ce6-65a47c117d64-kube-api-access-rl62c" (OuterVolumeSpecName: "kube-api-access-rl62c") pod "85ae65f7-e54f-487f-9ce6-65a47c117d64" (UID: "85ae65f7-e54f-487f-9ce6-65a47c117d64"). InnerVolumeSpecName "kube-api-access-rl62c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:38:32.992331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.992216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85ae65f7-e54f-487f-9ce6-65a47c117d64" (UID: "85ae65f7-e54f-487f-9ce6-65a47c117d64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:38:32.992660 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:32.992639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7f4345-b132-4b1b-a006-9e81755f362e-kube-api-access-qgthb" (OuterVolumeSpecName: "kube-api-access-qgthb") pod "1a7f4345-b132-4b1b-a006-9e81755f362e" (UID: "1a7f4345-b132-4b1b-a006-9e81755f362e"). InnerVolumeSpecName "kube-api-access-qgthb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:38:33.091046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091008    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/dc9251f4-0d28-479d-a2e1-a4602ad42435-kube-api-access-j9m88\") pod \"dc9251f4-0d28-479d-a2e1-a4602ad42435\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") "
Sep 14 10:38:33.091046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9251f4-0d28-479d-a2e1-a4602ad42435-plugins-config-volume\") pod \"dc9251f4-0d28-479d-a2e1-a4602ad42435\" (UID: \"dc9251f4-0d28-479d-a2e1-a4602ad42435\") "
Sep 14 10:38:33.091304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091196    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl62c\" (UniqueName: \"kubernetes.io/projected/85ae65f7-e54f-487f-9ce6-65a47c117d64-kube-api-access-rl62c\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:33.091304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091209    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:33.091304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091217    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ae65f7-e54f-487f-9ce6-65a47c117d64-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:33.091304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091225    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ae65f7-e54f-487f-9ce6-65a47c117d64-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:33.091304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091233    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgthb\" (UniqueName: \"kubernetes.io/projected/1a7f4345-b132-4b1b-a006-9e81755f362e-kube-api-access-qgthb\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:33.091550 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.091519    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9251f4-0d28-479d-a2e1-a4602ad42435-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc9251f4-0d28-479d-a2e1-a4602ad42435" (UID: "dc9251f4-0d28-479d-a2e1-a4602ad42435"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:38:33.093316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.093287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9251f4-0d28-479d-a2e1-a4602ad42435-kube-api-access-j9m88" (OuterVolumeSpecName: "kube-api-access-j9m88") pod "dc9251f4-0d28-479d-a2e1-a4602ad42435" (UID: "dc9251f4-0d28-479d-a2e1-a4602ad42435"). InnerVolumeSpecName "kube-api-access-j9m88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:38:33.192316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.192206    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/dc9251f4-0d28-479d-a2e1-a4602ad42435-kube-api-access-j9m88\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:33.192316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.192236    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc9251f4-0d28-479d-a2e1-a4602ad42435-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:33.456514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.456418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:33.763910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.763810    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a7f4345-b132-4b1b-a006-9e81755f362e" containerID="778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b" exitCode=0
Sep 14 10:38:33.763910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.763873    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"
Sep 14 10:38:33.764392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.763996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg" event={"ID":"1a7f4345-b132-4b1b-a006-9e81755f362e","Type":"ContainerDied","Data":"778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b"}
Sep 14 10:38:33.764392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.764078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg" event={"ID":"1a7f4345-b132-4b1b-a006-9e81755f362e","Type":"ContainerDied","Data":"5c06daafb9944000f75c4626913794e8646fa384246b227cd3f71629aaee6f90"}
Sep 14 10:38:33.764392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.764107    2563 scope.go:117] "RemoveContainer" containerID="778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b"
Sep 14 10:38:33.767784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.767761    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc9251f4-0d28-479d-a2e1-a4602ad42435" containerID="dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654" exitCode=0
Sep 14 10:38:33.767903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.767858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8" event={"ID":"dc9251f4-0d28-479d-a2e1-a4602ad42435","Type":"ContainerDied","Data":"dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654"}
Sep 14 10:38:33.767972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.767904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8" event={"ID":"dc9251f4-0d28-479d-a2e1-a4602ad42435","Type":"ContainerDied","Data":"9c0367ab45bd43928354464fe2be3da9fe2cc46004585943200a1888ae9114a9"}
Sep 14 10:38:33.768027 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.767968    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"
Sep 14 10:38:33.769418 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.769400    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"
Sep 14 10:38:33.769564 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.769442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz" event={"ID":"85ae65f7-e54f-487f-9ce6-65a47c117d64","Type":"ContainerDied","Data":"e6b7531dd0534656debc83e7372766984007c2ab57f1dd9f37ae52a9f2936e9f"}
Sep 14 10:38:33.777226 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.777208    2563 scope.go:117] "RemoveContainer" containerID="778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b"
Sep 14 10:38:33.777698 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:38:33.777516    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b\": container with ID starting with 778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b not found: ID does not exist" containerID="778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b"
Sep 14 10:38:33.777698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.777547    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b"} err="failed to get container status \"778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b\": rpc error: code = NotFound desc = could not find container \"778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b\": container with ID starting with 778c2f6ddebacdd04876edaf2222b313bb657d0ef778342ceea7ae1db2fecc1b not found: ID does not exist"
Sep 14 10:38:33.777698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.777597    2563 scope.go:117] "RemoveContainer" containerID="dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654"
Sep 14 10:38:33.786602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.786559    2563 scope.go:117] "RemoveContainer" containerID="dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654"
Sep 14 10:38:33.787099 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:38:33.787075    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654\": container with ID starting with dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654 not found: ID does not exist" containerID="dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654"
Sep 14 10:38:33.787184 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.787104    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654"} err="failed to get container status \"dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654\": rpc error: code = NotFound desc = could not find container \"dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654\": container with ID starting with dd2aa2216351fa8ab3559c086f9cb9096060d4954a8a44f64363e070303ce654 not found: ID does not exist"
Sep 14 10:38:33.787184 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.787128    2563 scope.go:117] "RemoveContainer" containerID="366f55d70414eac8cf382749da9b5fdd0594327a6904e1d336f6552de541c716"
Sep 14 10:38:33.788483 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.788465    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"]
Sep 14 10:38:33.794718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.794691    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-18c04da2-7b7f85b765-x27rg"]
Sep 14 10:38:33.804892 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.804869    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"]
Sep 14 10:38:33.807752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.807729    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-18c04da2-567cfcc759-v6vrz"]
Sep 14 10:38:33.816809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.816787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"]
Sep 14 10:38:33.821449 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.821425    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-18c04da2-647697cf94-t7jx8"]
Sep 14 10:38:33.919059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.919028    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7f4345-b132-4b1b-a006-9e81755f362e" path="/var/lib/kubelet/pods/1a7f4345-b132-4b1b-a006-9e81755f362e/volumes"
Sep 14 10:38:33.919411 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.919389    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ae65f7-e54f-487f-9ce6-65a47c117d64" path="/var/lib/kubelet/pods/85ae65f7-e54f-487f-9ce6-65a47c117d64/volumes"
Sep 14 10:38:33.919747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:33.919733    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc9251f4-0d28-479d-a2e1-a4602ad42435" path="/var/lib/kubelet/pods/dc9251f4-0d28-479d-a2e1-a4602ad42435/volumes"
Sep 14 10:38:34.022374 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.022281    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"]
Sep 14 10:38:34.049232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.049202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:34.325196 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.325106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:34.487304 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.487270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:34.777052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.777018    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy" containerID="cri-o://ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff" gracePeriod=30
Sep 14 10:38:34.872624 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.872593    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"]
Sep 14 10:38:34.872988 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.872976    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc9251f4-0d28-479d-a2e1-a4602ad42435" containerName="payload-pre-processing"
Sep 14 10:38:34.873030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.872989    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc9251f4-0d28-479d-a2e1-a4602ad42435" containerName="payload-pre-processing"
Sep 14 10:38:34.873030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.873010    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7f4345-b132-4b1b-a006-9e81755f362e" containerName="payload-processing"
Sep 14 10:38:34.873030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.873015    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7f4345-b132-4b1b-a006-9e81755f362e" containerName="payload-processing"
Sep 14 10:38:34.873030 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.873027    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ae65f7-e54f-487f-9ce6-65a47c117d64" containerName="maas-api"
Sep 14 10:38:34.873152 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.873033    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ae65f7-e54f-487f-9ce6-65a47c117d64" containerName="maas-api"
Sep 14 10:38:34.873152 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.873088    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7f4345-b132-4b1b-a006-9e81755f362e" containerName="payload-processing"
Sep 14 10:38:34.873152 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.873095    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ae65f7-e54f-487f-9ce6-65a47c117d64" containerName="maas-api"
Sep 14 10:38:34.873152 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.873105    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc9251f4-0d28-479d-a2e1-a4602ad42435" containerName="payload-pre-processing"
Sep 14 10:38:34.876003 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.875986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.878594 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.878554    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c62bec51-openshift-default-dockercfg-4wtxx\""
Sep 14 10:38:34.900104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.900075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"]
Sep 14 10:38:34.907848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.907820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-certs\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908004 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.907861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87k6m\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-kube-api-access-87k6m\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908004 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.907948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-socket\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908004 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.907998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb60bdff-e7b2-421c-b779-261414bc9b55-istiod-ca-cert\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.908028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-podinfo\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.908086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-credential-socket\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.908116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-envoy\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.908212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-token\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.908288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.908238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-data\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:34.931833 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:34.931801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:35.009329 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-token\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009329 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-data\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-certs\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87k6m\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-kube-api-access-87k6m\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-socket\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb60bdff-e7b2-421c-b779-261414bc9b55-istiod-ca-cert\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-podinfo\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-credential-socket\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-envoy\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-data\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-certs\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-socket\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.009977 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.009969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-credential-socket\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.010244 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.010218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb60bdff-e7b2-421c-b779-261414bc9b55-istiod-ca-cert\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.012137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.012110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-envoy\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.012264 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.012244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-podinfo\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.017645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.017616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87k6m\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-kube-api-access-87k6m\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.018128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.018103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-token\") pod \"e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") " pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.028006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.027897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:35.186633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.186597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:35.291889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.291807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:35.463132 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.463109    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"]
Sep 14 10:38:35.465647 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:38:35.465612    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb60bdff_e7b2_421c_b779_261414bc9b55.slice/crio-8ef326c1fd29af2cdfc8893d79238a047ae04fba1b6acab6a08087cd492960d2 WatchSource:0}: Error finding container 8ef326c1fd29af2cdfc8893d79238a047ae04fba1b6acab6a08087cd492960d2: Status 404 returned error can't find the container with id 8ef326c1fd29af2cdfc8893d79238a047ae04fba1b6acab6a08087cd492960d2
Sep 14 10:38:35.467887 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.467853    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:35.467982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.467922    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:35.467982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.467948    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:35.783962 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.783926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" event={"ID":"cb60bdff-e7b2-421c-b779-261414bc9b55","Type":"ContainerStarted","Data":"31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990"}
Sep 14 10:38:35.784325 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.783985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" event={"ID":"cb60bdff-e7b2-421c-b779-261414bc9b55","Type":"ContainerStarted","Data":"8ef326c1fd29af2cdfc8893d79238a047ae04fba1b6acab6a08087cd492960d2"}
Sep 14 10:38:35.806301 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:35.806249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podStartSLOduration=1.8062314800000001 podStartE2EDuration="1.80623148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:38:35.804663985 +0000 UTC m=+804.359851912" watchObservedRunningTime="2026-09-14 10:38:35.80623148 +0000 UTC m=+804.361419407"
Sep 14 10:38:36.102590 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:36.102475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:36.186797 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:36.186768    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:36.191410 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:36.191382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:36.780234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:36.780203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:36.788813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:36.788783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:36.789678 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:36.789658    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:37.243263 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:37.243228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:37.341377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:37.341346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:37.715434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:37.715378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:38.385068 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.385032    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"]
Sep 14 10:38:38.387780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.387759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.390088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.390065    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c62bec51-serving-cert\""
Sep 14 10:38:38.390200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.390181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:38:38.390419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.390378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bhlx\""
Sep 14 10:38:38.390519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.390471    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 10:38:38.390822 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.390804    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 10:38:38.400066 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.400044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"]
Sep 14 10:38:38.422333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.422304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:38.441582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.441535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4km\" (UniqueName: \"kubernetes.io/projected/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-kube-api-access-tx4km\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.441745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.441636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-maas-api-tls\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.441745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.441657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-openshift-service-ca\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.441745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.441703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-metrics-certs\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.517145 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.517114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:38.542822 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.542785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-metrics-certs\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.543017 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.542949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4km\" (UniqueName: \"kubernetes.io/projected/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-kube-api-access-tx4km\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.543017 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.542996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-maas-api-tls\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.543131 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.543024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-openshift-service-ca\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.543753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.543671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-openshift-service-ca\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.545433 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.545412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-maas-api-tls\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.545530 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.545459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-metrics-certs\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.551115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.551094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4km\" (UniqueName: \"kubernetes.io/projected/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-kube-api-access-tx4km\") pod \"maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.618269 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.618235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:38.700062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.699981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:38.986422 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.986392    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"]
Sep 14 10:38:38.986670 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:38.986636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:38.988160 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:38:38.988126    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0d67db_87fa_4e68_a8e4_dfb263f70cb9.slice/crio-f265217804e0c3b5b20715e1c7f4b142a710316f6d7a5c43a4573c704e085795 WatchSource:0}: Error finding container f265217804e0c3b5b20715e1c7f4b142a710316f6d7a5c43a4573c704e085795: Status 404 returned error can't find the container with id f265217804e0c3b5b20715e1c7f4b142a710316f6d7a5c43a4573c704e085795
Sep 14 10:38:39.097865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.097839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:39.216439 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.216403    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"]
Sep 14 10:38:39.219786 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.219765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.221719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.221698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c62bec51\""
Sep 14 10:38:39.221818 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.221761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c62bec51-dockercfg-l2fj2\""
Sep 14 10:38:39.227953 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.227928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"]
Sep 14 10:38:39.352415 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.352334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nk6t\" (UniqueName: \"kubernetes.io/projected/fdfc07ae-5290-49a4-a9de-3aae167ade45-kube-api-access-7nk6t\") pod \"payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.352415 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.352405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfc07ae-5290-49a4-a9de-3aae167ade45-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.453466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.453429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nk6t\" (UniqueName: \"kubernetes.io/projected/fdfc07ae-5290-49a4-a9de-3aae167ade45-kube-api-access-7nk6t\") pod \"payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.453861 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.453504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfc07ae-5290-49a4-a9de-3aae167ade45-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.454224 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.454204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfc07ae-5290-49a4-a9de-3aae167ade45-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.461448 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.461427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nk6t\" (UniqueName: \"kubernetes.io/projected/fdfc07ae-5290-49a4-a9de-3aae167ade45-kube-api-access-7nk6t\") pod \"payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.531372 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.531346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:39.571104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.571070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:39.804322 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.804243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"]
Sep 14 10:38:39.807984 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.807961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g" event={"ID":"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9","Type":"ContainerStarted","Data":"d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7"}
Sep 14 10:38:39.808074 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.808003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g" event={"ID":"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9","Type":"ContainerStarted","Data":"f265217804e0c3b5b20715e1c7f4b142a710316f6d7a5c43a4573c704e085795"}
Sep 14 10:38:39.808114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.808078    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:39.827139 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:39.827085    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g" podStartSLOduration=1.3457613290000001 podStartE2EDuration="1.827066398s" totalImagesPullingTime="481.305069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:38:39.823938382 +0000 UTC m=+808.379126344" watchObservedRunningTime="2026-09-14 10:38:39.827066398 +0000 UTC m=+808.382254325"
Sep 14 10:38:39.838396 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:38:39.838365    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfc07ae_5290_49a4_a9de_3aae167ade45.slice/crio-b42acdc58ed32dd97229b6f66046ca7a5d04cbf841c949e2879653a1aa736d3e WatchSource:0}: Error finding container b42acdc58ed32dd97229b6f66046ca7a5d04cbf841c949e2879653a1aa736d3e: Status 404 returned error can't find the container with id b42acdc58ed32dd97229b6f66046ca7a5d04cbf841c949e2879653a1aa736d3e
Sep 14 10:38:40.017213 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.017188    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:40.058959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.058878    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d388bf6b-072a-46a8-8066-7f18368bb919-istiod-ca-cert\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.058959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.058919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-socket\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.058959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.058940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-credential-socket\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.059200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.058994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-istio-token\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.059200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059036    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-certs\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.059200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-data\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.059200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d388bf6b-072a-46a8-8066-7f18368bb919-istio-podinfo\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.059200 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2578v\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-kube-api-access-2578v\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.059434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-envoy\") pod \"d388bf6b-072a-46a8-8066-7f18368bb919\" (UID: \"d388bf6b-072a-46a8-8066-7f18368bb919\") "
Sep 14 10:38:40.059559 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:40.059559 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:40.059559 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059341    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d388bf6b-072a-46a8-8066-7f18368bb919-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:38:40.060011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-data" (OuterVolumeSpecName: "istio-data") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:40.060011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.059701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:40.061278 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.061252    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-istio-token" (OuterVolumeSpecName: "istio-token") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:38:40.061348 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.061320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-kube-api-access-2578v" (OuterVolumeSpecName: "kube-api-access-2578v") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "kube-api-access-2578v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:38:40.061793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.061777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d388bf6b-072a-46a8-8066-7f18368bb919-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:38:40.061847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.061822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d388bf6b-072a-46a8-8066-7f18368bb919" (UID: "d388bf6b-072a-46a8-8066-7f18368bb919"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:38:40.160985 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.160956    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d388bf6b-072a-46a8-8066-7f18368bb919-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.160985 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.160982    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.160985 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.160992    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.161255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.161002    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.161255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.161011    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.161255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.161020    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.161255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.161032    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d388bf6b-072a-46a8-8066-7f18368bb919-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.161255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.161042    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2578v\" (UniqueName: \"kubernetes.io/projected/d388bf6b-072a-46a8-8066-7f18368bb919-kube-api-access-2578v\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.161255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.161049    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d388bf6b-072a-46a8-8066-7f18368bb919-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:38:40.282709 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.282660    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:40.815565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.815467    2563 generic.go:356] "Generic (PLEG): container finished" podID="d388bf6b-072a-46a8-8066-7f18368bb919" containerID="ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff" exitCode=0
Sep 14 10:38:40.815565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.815522    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"
Sep 14 10:38:40.816110 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.815558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" event={"ID":"d388bf6b-072a-46a8-8066-7f18368bb919","Type":"ContainerDied","Data":"ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff"}
Sep 14 10:38:40.816110 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.815648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r" event={"ID":"d388bf6b-072a-46a8-8066-7f18368bb919","Type":"ContainerDied","Data":"f4bd6e0de9d3d89b80feb593a6e91e8a3cc99da429ccb7c043468e58de7bc040"}
Sep 14 10:38:40.816110 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.815680    2563 scope.go:117] "RemoveContainer" containerID="ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff"
Sep 14 10:38:40.817434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.817400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6" event={"ID":"fdfc07ae-5290-49a4-a9de-3aae167ade45","Type":"ContainerStarted","Data":"282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d"}
Sep 14 10:38:40.817583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.817457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6" event={"ID":"fdfc07ae-5290-49a4-a9de-3aae167ade45","Type":"ContainerStarted","Data":"b42acdc58ed32dd97229b6f66046ca7a5d04cbf841c949e2879653a1aa736d3e"}
Sep 14 10:38:40.817583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.817489    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:40.825547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.825530    2563 scope.go:117] "RemoveContainer" containerID="dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe"
Sep 14 10:38:40.833639 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.833623    2563 scope.go:117] "RemoveContainer" containerID="ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff"
Sep 14 10:38:40.833857 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:38:40.833838    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff\": container with ID starting with ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff not found: ID does not exist" containerID="ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff"
Sep 14 10:38:40.833921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.833870    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff"} err="failed to get container status \"ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff\": rpc error: code = NotFound desc = could not find container \"ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff\": container with ID starting with ebb9c9b77eafef77c1777acbbce5495a45096f72f015768546439babe6d503ff not found: ID does not exist"
Sep 14 10:38:40.833921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.833896    2563 scope.go:117] "RemoveContainer" containerID="dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe"
Sep 14 10:38:40.834172 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:38:40.834156    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe\": container with ID starting with dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe not found: ID does not exist" containerID="dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe"
Sep 14 10:38:40.834222 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.834178    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe"} err="failed to get container status \"dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe\": rpc error: code = NotFound desc = could not find container \"dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe\": container with ID starting with dcd000f2184c4dae53e7adba2adc5b0d82d1b855e224d751e397397408dcd2fe not found: ID does not exist"
Sep 14 10:38:40.835792 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.835729    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6" podStartSLOduration=1.8357132360000001 podStartE2EDuration="1.835713236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:38:40.835391307 +0000 UTC m=+809.390579232" watchObservedRunningTime="2026-09-14 10:38:40.835713236 +0000 UTC m=+809.390901157"
Sep 14 10:38:40.852766 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.852727    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"]
Sep 14 10:38:40.859008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:40.858981    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-18c04da2-openshift-default-555c79855d-vth4r"]
Sep 14 10:38:41.089088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:41.089010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:41.919280 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:41.919248    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" path="/var/lib/kubelet/pods/d388bf6b-072a-46a8-8066-7f18368bb919/volumes"
Sep 14 10:38:41.999631 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:41.999590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:42.644190 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:42.644149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:44.767426 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:44.767389    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:38:45.822770 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:45.822740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:38:48.059281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.059244    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"]
Sep 14 10:38:48.059697 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.059667    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy"
Sep 14 10:38:48.059697 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.059679    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy"
Sep 14 10:38:48.059776 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.059753    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy"
Sep 14 10:38:48.062982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.062959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.065193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.065163    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d36f67\"/\"openshift-service-ca.crt\""
Sep 14 10:38:48.065292 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.065228    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d36f67\"/\"kube-root-ca.crt\""
Sep 14 10:38:48.065800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.065783    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d36f67\"/\"test-model-d36f67-kserve-self-signed-certs\""
Sep 14 10:38:48.065858 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.065807    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d36f67\"/\"default-dockercfg-8qskt\""
Sep 14 10:38:48.073461 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.073440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"]
Sep 14 10:38:48.080352 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.080329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:48.143171 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.143132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tmp-dir\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.143352 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.143225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tls-certs\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.143352 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.143297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-home\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.143424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.143349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/ad047ad2-0dfe-40b1-a155-33a6f609bd92-kube-api-access-q2dzr\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.143424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.143393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-dshm\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.143492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.143424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-model-cache\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.244533 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.244498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-home\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.244722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.244558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/ad047ad2-0dfe-40b1-a155-33a6f609bd92-kube-api-access-q2dzr\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.244722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.244614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-dshm\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.244722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.244642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-model-cache\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.244722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.244689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tmp-dir\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.244958 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.244791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tls-certs\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.244958 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.244945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-home\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.245069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.245041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-model-cache\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.245109 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.245088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tmp-dir\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.246996 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.246974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-dshm\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.247234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.247218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tls-certs\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.252790 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.252768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/ad047ad2-0dfe-40b1-a155-33a6f609bd92-kube-api-access-q2dzr\") pod \"test-model-d36f67-kserve-77477589f7-mjbp6\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") " pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.307137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.307109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:48.373840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.373809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:48.408605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.408552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:48.665803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.665725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"]
Sep 14 10:38:48.668911 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:38:48.668883    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad047ad2_0dfe_40b1_a155_33a6f609bd92.slice/crio-ee1b32cd4f224228bd6e7da010394037e1827350f60e8f2b1ca3ac1d032b6692 WatchSource:0}: Error finding container ee1b32cd4f224228bd6e7da010394037e1827350f60e8f2b1ca3ac1d032b6692: Status 404 returned error can't find the container with id ee1b32cd4f224228bd6e7da010394037e1827350f60e8f2b1ca3ac1d032b6692
Sep 14 10:38:48.865655 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.865622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6" event={"ID":"ad047ad2-0dfe-40b1-a155-33a6f609bd92","Type":"ContainerStarted","Data":"26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925"}
Sep 14 10:38:48.865818 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.865664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6" event={"ID":"ad047ad2-0dfe-40b1-a155-33a6f609bd92","Type":"ContainerStarted","Data":"ee1b32cd4f224228bd6e7da010394037e1827350f60e8f2b1ca3ac1d032b6692"}
Sep 14 10:38:48.884038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:48.883985    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6" podStartSLOduration=0.883970524 podStartE2EDuration="883.970524ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:38:48.882878487 +0000 UTC m=+817.438066412" watchObservedRunningTime="2026-09-14 10:38:48.883970524 +0000 UTC m=+817.439158494"
Sep 14 10:38:51.828515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:51.828415    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:38:52.830784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:52.830740    2563 patch_prober.go:28] interesting pod/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:38:52.831158 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:52.830820    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:38:54.899766 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:54.899728    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerID="31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990" exitCode=0
Sep 14 10:38:54.900167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:54.899801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" event={"ID":"cb60bdff-e7b2-421c-b779-261414bc9b55","Type":"ContainerDied","Data":"31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990"}
Sep 14 10:38:54.900225 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:54.900210    2563 scope.go:117] "RemoveContainer" containerID="31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990"
Sep 14 10:38:54.901337 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:54.901308    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:54.901400 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:54.901377    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:54.901436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:54.901415    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:38:55.907951 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:55.907915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" event={"ID":"cb60bdff-e7b2-421c-b779-261414bc9b55","Type":"ContainerStarted","Data":"efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0"}
Sep 14 10:38:56.187263 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:56.187171    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:38:56.187418 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:56.187400    2563 patch_prober.go:28] interesting pod/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 10:38:56.187506 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:56.187482    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 10:38:57.187107 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:57.187063    2563 patch_prober.go:28] interesting pod/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 10:38:57.187524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:57.187133    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 10:38:58.187958 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:58.187925    2563 patch_prober.go:28] interesting pod/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 10:38:58.188346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:58.188003    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 10:38:58.374670 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:58.374630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:58.374908 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:58.374876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:58.387061 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:58.387039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:58.916766 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:58.916697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:38:58.946804 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:58.946777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:38:59.187420 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:59.187320    2563 patch_prober.go:28] interesting pod/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 10:38:59.187420 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:59.187383    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 10:38:59.737463 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:38:59.737426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:00.187359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:00.187327    2563 patch_prober.go:28] interesting pod/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 10:39:00.187533 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:00.187382    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 10:39:00.569312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:00.569220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:01.187403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:01.187362    2563 patch_prober.go:28] interesting pod/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 14 10:39:01.187824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:01.187429    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 14 10:39:01.423107 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:01.423064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:02.191125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:02.191090    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:39:02.191496 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:02.191367    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:39:02.192061 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:02.192042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:39:02.410475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:02.410446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:03.273495 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:03.273457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:03.921679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:03.921646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:04.695894 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:04.695859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:05.360151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:05.360119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:06.026295 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:06.026262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:06.807340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:06.807299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:06.866008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:06.865969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:39:07.484382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:07.484347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:08.804044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:08.804009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.369556 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.369514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.404302 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.404266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"]
Sep 14 10:39:09.404540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.404492    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf" podUID="b0f383d4-f629-4641-89c7-8fd696d15372" containerName="payload-pre-processing" containerID="cri-o://d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038" gracePeriod=30
Sep 14 10:39:09.462472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.462440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.522255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.522221    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"]
Sep 14 10:39:09.522453 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.522430    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g" podUID="7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" containerName="maas-api" containerID="cri-o://d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7" gracePeriod=30
Sep 14 10:39:09.549103 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.549059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.587725 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.587684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"]
Sep 14 10:39:09.587904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.587881    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6" podUID="fdfc07ae-5290-49a4-a9de-3aae167ade45" containerName="payload-pre-processing" containerID="cri-o://282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d" gracePeriod=30
Sep 14 10:39:09.628554 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.628516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.710032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.710008    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:39:09.723776 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.723746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.766707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.766318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbpgw\" (UniqueName: \"kubernetes.io/projected/b0f383d4-f629-4641-89c7-8fd696d15372-kube-api-access-bbpgw\") pod \"b0f383d4-f629-4641-89c7-8fd696d15372\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") "
Sep 14 10:39:09.766707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.766403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f383d4-f629-4641-89c7-8fd696d15372-plugins-config-volume\") pod \"b0f383d4-f629-4641-89c7-8fd696d15372\" (UID: \"b0f383d4-f629-4641-89c7-8fd696d15372\") "
Sep 14 10:39:09.767420 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.767165    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f383d4-f629-4641-89c7-8fd696d15372-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0f383d4-f629-4641-89c7-8fd696d15372" (UID: "b0f383d4-f629-4641-89c7-8fd696d15372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:39:09.769777 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.769750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f383d4-f629-4641-89c7-8fd696d15372-kube-api-access-bbpgw" (OuterVolumeSpecName: "kube-api-access-bbpgw") pod "b0f383d4-f629-4641-89c7-8fd696d15372" (UID: "b0f383d4-f629-4641-89c7-8fd696d15372"). InnerVolumeSpecName "kube-api-access-bbpgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:39:09.823516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.823476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.831698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.831673    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:39:09.867064 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-openshift-service-ca\") pod \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") "
Sep 14 10:39:09.867233 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-maas-api-tls\") pod \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") "
Sep 14 10:39:09.867233 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-metrics-certs\") pod \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") "
Sep 14 10:39:09.867347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867245    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx4km\" (UniqueName: \"kubernetes.io/projected/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-kube-api-access-tx4km\") pod \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\" (UID: \"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9\") "
Sep 14 10:39:09.867395 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867375    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" (UID: "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:39:09.867669 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867647    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:09.867669 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867671    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpgw\" (UniqueName: \"kubernetes.io/projected/b0f383d4-f629-4641-89c7-8fd696d15372-kube-api-access-bbpgw\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:09.867866 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.867683    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f383d4-f629-4641-89c7-8fd696d15372-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:09.868679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.868609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:39:09.869389 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.869359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" (UID: "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:39:09.869484 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.869378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" (UID: "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:39:09.869697 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.869675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-kube-api-access-tx4km" (OuterVolumeSpecName: "kube-api-access-tx4km") pod "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" (UID: "7a0d67db-87fa-4e68-a8e4-dfb263f70cb9"). InnerVolumeSpecName "kube-api-access-tx4km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:39:09.900763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.900674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.969051 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.969013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfc07ae-5290-49a4-a9de-3aae167ade45-plugins-config-volume\") pod \"fdfc07ae-5290-49a4-a9de-3aae167ade45\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") "
Sep 14 10:39:09.969260 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.969087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nk6t\" (UniqueName: \"kubernetes.io/projected/fdfc07ae-5290-49a4-a9de-3aae167ade45-kube-api-access-7nk6t\") pod \"fdfc07ae-5290-49a4-a9de-3aae167ade45\" (UID: \"fdfc07ae-5290-49a4-a9de-3aae167ade45\") "
Sep 14 10:39:09.969465 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.969438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfc07ae-5290-49a4-a9de-3aae167ade45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdfc07ae-5290-49a4-a9de-3aae167ade45" (UID: "fdfc07ae-5290-49a4-a9de-3aae167ade45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:39:09.969539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.969450    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:09.969539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.969500    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:09.969539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.969518    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx4km\" (UniqueName: \"kubernetes.io/projected/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9-kube-api-access-tx4km\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:09.973014 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.972988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfc07ae-5290-49a4-a9de-3aae167ade45-kube-api-access-7nk6t" (OuterVolumeSpecName: "kube-api-access-7nk6t") pod "fdfc07ae-5290-49a4-a9de-3aae167ade45" (UID: "fdfc07ae-5290-49a4-a9de-3aae167ade45"). InnerVolumeSpecName "kube-api-access-7nk6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:39:09.981223 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.981192    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"]
Sep 14 10:39:09.981472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.981436    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy" containerID="cri-o://efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0" gracePeriod=30
Sep 14 10:39:09.982821 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.982790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:09.999791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.999756    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdfc07ae-5290-49a4-a9de-3aae167ade45" containerID="282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d" exitCode=0
Sep 14 10:39:09.999932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.999790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6" event={"ID":"fdfc07ae-5290-49a4-a9de-3aae167ade45","Type":"ContainerDied","Data":"282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d"}
Sep 14 10:39:09.999932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.999820    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"
Sep 14 10:39:09.999932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.999843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6" event={"ID":"fdfc07ae-5290-49a4-a9de-3aae167ade45","Type":"ContainerDied","Data":"b42acdc58ed32dd97229b6f66046ca7a5d04cbf841c949e2879653a1aa736d3e"}
Sep 14 10:39:09.999932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:09.999872    2563 scope.go:117] "RemoveContainer" containerID="282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d"
Sep 14 10:39:10.002252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.002187    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" containerID="d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7" exitCode=0
Sep 14 10:39:10.002252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.002233    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"
Sep 14 10:39:10.002252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.002262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g" event={"ID":"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9","Type":"ContainerDied","Data":"d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7"}
Sep 14 10:39:10.002512 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.002331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g" event={"ID":"7a0d67db-87fa-4e68-a8e4-dfb263f70cb9","Type":"ContainerDied","Data":"f265217804e0c3b5b20715e1c7f4b142a710316f6d7a5c43a4573c704e085795"}
Sep 14 10:39:10.003989 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.003956    2563 generic.go:356] "Generic (PLEG): container finished" podID="b0f383d4-f629-4641-89c7-8fd696d15372" containerID="d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038" exitCode=0
Sep 14 10:39:10.004101 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.003984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf" event={"ID":"b0f383d4-f629-4641-89c7-8fd696d15372","Type":"ContainerDied","Data":"d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038"}
Sep 14 10:39:10.004101 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.004022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf" event={"ID":"b0f383d4-f629-4641-89c7-8fd696d15372","Type":"ContainerDied","Data":"51bcc6fd8713eb19f58a94476f6a9bda16967ac52631fb8ee88c2d958f9db8e2"}
Sep 14 10:39:10.004101 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.004027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"
Sep 14 10:39:10.011455 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.011436    2563 scope.go:117] "RemoveContainer" containerID="282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d"
Sep 14 10:39:10.011823 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:39:10.011802    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d\": container with ID starting with 282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d not found: ID does not exist" containerID="282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d"
Sep 14 10:39:10.011892 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.011831    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d"} err="failed to get container status \"282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d\": rpc error: code = NotFound desc = could not find container \"282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d\": container with ID starting with 282faaae277de43d4b1debbb1d637363cba4fae849ce6b88f1180c4b1e364f0d not found: ID does not exist"
Sep 14 10:39:10.011892 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.011848    2563 scope.go:117] "RemoveContainer" containerID="d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7"
Sep 14 10:39:10.020891 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.020713    2563 scope.go:117] "RemoveContainer" containerID="d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7"
Sep 14 10:39:10.021098 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:39:10.021075    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7\": container with ID starting with d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7 not found: ID does not exist" containerID="d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7"
Sep 14 10:39:10.021167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.021110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7"} err="failed to get container status \"d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7\": rpc error: code = NotFound desc = could not find container \"d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7\": container with ID starting with d03e4f6af8e845ca22f94eca039a0e31ad45d10bae95091a6a054070ea71e5a7 not found: ID does not exist"
Sep 14 10:39:10.021167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.021137    2563 scope.go:117] "RemoveContainer" containerID="d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038"
Sep 14 10:39:10.023008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.022985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"]
Sep 14 10:39:10.027548 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.027528    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c62bec51-c8b96db46-t6n5g"]
Sep 14 10:39:10.030186 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.030164    2563 scope.go:117] "RemoveContainer" containerID="d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038"
Sep 14 10:39:10.030463 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:39:10.030442    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038\": container with ID starting with d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038 not found: ID does not exist" containerID="d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038"
Sep 14 10:39:10.030511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.030473    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038"} err="failed to get container status \"d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038\": rpc error: code = NotFound desc = could not find container \"d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038\": container with ID starting with d369dba6c399952fb1022d0da19ce9dcb444e4a9bb04f9b34ff63ee730cc7038 not found: ID does not exist"
Sep 14 10:39:10.041172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.041148    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"]
Sep 14 10:39:10.046800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.046776    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c62bec51-6677694ff4-5x6q6"]
Sep 14 10:39:10.058876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.058852    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"]
Sep 14 10:39:10.065816 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.065794    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-30eaba-56bb688b6b-tdlmf"]
Sep 14 10:39:10.070622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.070560    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfc07ae-5290-49a4-a9de-3aae167ade45-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:10.070622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.070606    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nk6t\" (UniqueName: \"kubernetes.io/projected/fdfc07ae-5290-49a4-a9de-3aae167ade45-kube-api-access-7nk6t\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:10.074943 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.074918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:10.176155 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.176073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:10.697627 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.697591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:10.794808 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.794774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:10.896059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.896015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:10.995303 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:10.995223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:11.764905 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:11.764871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:11.919464 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:11.919436    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" path="/var/lib/kubelet/pods/7a0d67db-87fa-4e68-a8e4-dfb263f70cb9/volumes"
Sep 14 10:39:11.919843 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:11.919829    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f383d4-f629-4641-89c7-8fd696d15372" path="/var/lib/kubelet/pods/b0f383d4-f629-4641-89c7-8fd696d15372/volumes"
Sep 14 10:39:11.920153 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:11.920141    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfc07ae-5290-49a4-a9de-3aae167ade45" path="/var/lib/kubelet/pods/fdfc07ae-5290-49a4-a9de-3aae167ade45/volumes"
Sep 14 10:39:12.282024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:12.281979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:12.618032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:12.617949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:13.314658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:13.314621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:14.073586 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:14.073531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:15.054545 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.054513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:15.228333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.228310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:39:15.320622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87k6m\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-kube-api-access-87k6m\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.320622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-token\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.320841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320647    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-credential-socket\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.320841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-socket\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.320841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb60bdff-e7b2-421c-b779-261414bc9b55-istiod-ca-cert\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.320841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-certs\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.320841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-podinfo\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.320841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320821    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-data\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.321135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-envoy\") pod \"cb60bdff-e7b2-421c-b779-261414bc9b55\" (UID: \"cb60bdff-e7b2-421c-b779-261414bc9b55\") "
Sep 14 10:39:15.321135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.320997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:39:15.321246 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.321133    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:39:15.321246 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.321171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb60bdff-e7b2-421c-b779-261414bc9b55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:39:15.321357 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.321293    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.321357 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.321312    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb60bdff-e7b2-421c-b779-261414bc9b55-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.321357 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.321325    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.321357 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.321325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:39:15.321539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.321442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:39:15.322858 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.322832    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:39:15.322972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.322937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-kube-api-access-87k6m" (OuterVolumeSpecName: "kube-api-access-87k6m") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "kube-api-access-87k6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:39:15.323061 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.323044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:39:15.323310 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.323292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb60bdff-e7b2-421c-b779-261414bc9b55" (UID: "cb60bdff-e7b2-421c-b779-261414bc9b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:39:15.422633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.422607    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.422749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.422640    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.422749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.422649    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.422749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.422658    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.422749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.422666    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb60bdff-e7b2-421c-b779-261414bc9b55-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:15.422749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:15.422674    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87k6m\" (UniqueName: \"kubernetes.io/projected/cb60bdff-e7b2-421c-b779-261414bc9b55-kube-api-access-87k6m\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:39:16.038928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.038899    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerID="efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0" exitCode=0
Sep 14 10:39:16.039065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.038938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" event={"ID":"cb60bdff-e7b2-421c-b779-261414bc9b55","Type":"ContainerDied","Data":"efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0"}
Sep 14 10:39:16.039065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.038953    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"
Sep 14 10:39:16.039065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.038976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm" event={"ID":"cb60bdff-e7b2-421c-b779-261414bc9b55","Type":"ContainerDied","Data":"8ef326c1fd29af2cdfc8893d79238a047ae04fba1b6acab6a08087cd492960d2"}
Sep 14 10:39:16.039065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.038999    2563 scope.go:117] "RemoveContainer" containerID="efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0"
Sep 14 10:39:16.047816 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.047796    2563 scope.go:117] "RemoveContainer" containerID="31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990"
Sep 14 10:39:16.055740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.055720    2563 scope.go:117] "RemoveContainer" containerID="efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0"
Sep 14 10:39:16.056046 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:39:16.055970    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0\": container with ID starting with efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0 not found: ID does not exist" containerID="efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0"
Sep 14 10:39:16.056046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.055997    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0"} err="failed to get container status \"efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0\": rpc error: code = NotFound desc = could not find container \"efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0\": container with ID starting with efb52ee334cf65a647f212fd265f5b58eed9a3d7e5a9dd8d38970e50f50a5ec0 not found: ID does not exist"
Sep 14 10:39:16.056046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.056015    2563 scope.go:117] "RemoveContainer" containerID="31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990"
Sep 14 10:39:16.056046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.056003    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"]
Sep 14 10:39:16.056237 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:39:16.056221    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990\": container with ID starting with 31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990 not found: ID does not exist" containerID="31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990"
Sep 14 10:39:16.056281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.056243    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990"} err="failed to get container status \"31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990\": rpc error: code = NotFound desc = could not find container \"31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990\": container with ID starting with 31bb71d6ef1d4028eacacb027be3a540a800e6f41f267a96535501c5fa960990 not found: ID does not exist"
Sep 14 10:39:16.062160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:16.062139    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c62bec51-openshift-default-585f889dfb-mx7jm"]
Sep 14 10:39:17.918137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:17.918102    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" path="/var/lib/kubelet/pods/cb60bdff-e7b2-421c-b779-261414bc9b55/volumes"
Sep 14 10:39:18.074434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:18.074407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:18.134584 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:18.134548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:18.234735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:18.234642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:18.468957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:18.468925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:18.799477 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:18.799442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:25.800342 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:25.800301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:27.965234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:27.965203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:39:34.671956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.671920    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"]
Sep 14 10:39:34.672511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672483    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfc07ae-5290-49a4-a9de-3aae167ade45" containerName="payload-pre-processing"
Sep 14 10:39:34.672511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672502    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfc07ae-5290-49a4-a9de-3aae167ade45" containerName="payload-pre-processing"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672524    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f383d4-f629-4641-89c7-8fd696d15372" containerName="payload-pre-processing"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672532    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f383d4-f629-4641-89c7-8fd696d15372" containerName="payload-pre-processing"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" containerName="maas-api"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672554    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" containerName="maas-api"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672603    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672613    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672623    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672630    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672640    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy"
Sep 14 10:39:34.672679 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672648    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy"
Sep 14 10:39:34.673142 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672750    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0d67db-87fa-4e68-a8e4-dfb263f70cb9" containerName="maas-api"
Sep 14 10:39:34.673142 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672764    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f383d4-f629-4641-89c7-8fd696d15372" containerName="payload-pre-processing"
Sep 14 10:39:34.673142 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672773    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d388bf6b-072a-46a8-8066-7f18368bb919" containerName="istio-proxy"
Sep 14 10:39:34.673142 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672782    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfc07ae-5290-49a4-a9de-3aae167ade45" containerName="payload-pre-processing"
Sep 14 10:39:34.673142 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672795    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy"
Sep 14 10:39:34.673142 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.672803    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb60bdff-e7b2-421c-b779-261414bc9b55" containerName="istio-proxy"
Sep 14 10:39:34.676453 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.676434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.678477 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.678453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-85d3ec\"/\"premium-w0-85d3ec-kserve-self-signed-certs\""
Sep 14 10:39:34.692428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.690037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"]
Sep 14 10:39:34.766377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.766344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:34.800336 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.800309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a47ee3-35ae-4602-a460-3d8f69d188ec-tls-certs\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.800452 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.800348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-model-cache\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.800452 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.800434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-home\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.800522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.800455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-tmp-dir\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.800522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.800474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-dshm\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.800522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.800505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vzh\" (UniqueName: \"kubernetes.io/projected/84a47ee3-35ae-4602-a460-3d8f69d188ec-kube-api-access-z2vzh\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.900991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-home\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-tmp-dir\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-dshm\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vzh\" (UniqueName: \"kubernetes.io/projected/84a47ee3-35ae-4602-a460-3d8f69d188ec-kube-api-access-z2vzh\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901380 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a47ee3-35ae-4602-a460-3d8f69d188ec-tls-certs\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901380 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-model-cache\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901520 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-tmp-dir\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901628 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-home\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.901700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.901662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-model-cache\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.903453 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.903423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-dshm\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.903859 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.903842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a47ee3-35ae-4602-a460-3d8f69d188ec-tls-certs\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.908907 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.908891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vzh\" (UniqueName: \"kubernetes.io/projected/84a47ee3-35ae-4602-a460-3d8f69d188ec-kube-api-access-z2vzh\") pod \"premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") " pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:34.991614 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:34.991538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:35.259860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:35.259837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"]
Sep 14 10:39:35.261904 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:39:35.261875    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84a47ee3_35ae_4602_a460_3d8f69d188ec.slice/crio-8294bdfbe1a11f7cfb9aac828c01178d830a927ad7c71ab2925c28da4c6313de WatchSource:0}: Error finding container 8294bdfbe1a11f7cfb9aac828c01178d830a927ad7c71ab2925c28da4c6313de: Status 404 returned error can't find the container with id 8294bdfbe1a11f7cfb9aac828c01178d830a927ad7c71ab2925c28da4c6313de
Sep 14 10:39:36.140946 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:36.140913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk" event={"ID":"84a47ee3-35ae-4602-a460-3d8f69d188ec","Type":"ContainerStarted","Data":"3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c"}
Sep 14 10:39:36.141282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:36.140956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk" event={"ID":"84a47ee3-35ae-4602-a460-3d8f69d188ec","Type":"ContainerStarted","Data":"8294bdfbe1a11f7cfb9aac828c01178d830a927ad7c71ab2925c28da4c6313de"}
Sep 14 10:39:36.159698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:36.159647    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk" podStartSLOduration=2.159632767 podStartE2EDuration="2.159632767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:39:36.158119912 +0000 UTC m=+864.713307839" watchObservedRunningTime="2026-09-14 10:39:36.159632767 +0000 UTC m=+864.714820692"
Sep 14 10:39:41.765900 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:41.765862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:39:44.734650 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:44.734600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:44.992067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:44.991987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:44.992067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:44.992045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:45.004089 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.004063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:45.046696 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.046670    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"]
Sep 14 10:39:45.049296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.049279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.051027 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.051009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1910e9\"/\"premium-w4-1910e9-kserve-self-signed-certs\""
Sep 14 10:39:45.062234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.062212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"]
Sep 14 10:39:45.076127 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.076095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-model-cache\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.076127 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.076123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-dshm\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.076302 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.076152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-kube-api-access-gfth8\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.076302 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.076224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-home\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.076417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.076316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tls-certs\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.076417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.076349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tmp-dir\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-kube-api-access-gfth8\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177113 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-home\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177113 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tls-certs\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177196 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tmp-dir\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177278 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-model-cache\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177323 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-dshm\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177506 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tmp-dir\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177543 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-home\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.177595 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.177542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-model-cache\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.179427 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.179401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-dshm\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.179720 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.179699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tls-certs\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.185533 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.185508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-kube-api-access-gfth8\") pod \"premium-w4-1910e9-kserve-74d75945b7-m574p\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") " pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.195711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.195691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:39:45.260136 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.260025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:45.358634 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.358610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:45.629682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:45.629660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"]
Sep 14 10:39:45.631992 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:39:45.631964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedaecc21_86b5_4e62_bcdd_90076b9ee8fd.slice/crio-f4334b1820e107d3c8a5ed0a5768827aa1b5de6865d922e7b730a460439078f2 WatchSource:0}: Error finding container f4334b1820e107d3c8a5ed0a5768827aa1b5de6865d922e7b730a460439078f2: Status 404 returned error can't find the container with id f4334b1820e107d3c8a5ed0a5768827aa1b5de6865d922e7b730a460439078f2
Sep 14 10:39:46.192762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:46.192721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p" event={"ID":"edaecc21-86b5-4e62-bcdd-90076b9ee8fd","Type":"ContainerStarted","Data":"24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0"}
Sep 14 10:39:46.192762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:46.192767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p" event={"ID":"edaecc21-86b5-4e62-bcdd-90076b9ee8fd","Type":"ContainerStarted","Data":"f4334b1820e107d3c8a5ed0a5768827aa1b5de6865d922e7b730a460439078f2"}
Sep 14 10:39:46.211518 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:46.211472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p" podStartSLOduration=1.211457858 podStartE2EDuration="1.211457858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:39:46.20911149 +0000 UTC m=+874.764299418" watchObservedRunningTime="2026-09-14 10:39:46.211457858 +0000 UTC m=+874.766645782"
Sep 14 10:39:50.719173 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:50.719145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:50.819938 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:50.819892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:50.918754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:50.918724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:52.360165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:52.360130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:52.636532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:52.636489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:53.085633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.085535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:53.431159 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.431117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:53.492469 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.492431    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"]
Sep 14 10:39:53.495875 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.495852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.498146 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.498127    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 10:39:53.498600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.498559    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-cc7bcd-openshift-default-dockercfg-mr48w\""
Sep 14 10:39:53.517419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.517398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"]
Sep 14 10:39:53.529039 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.529001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:53.549889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.549867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istiod-ca-cert\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.549991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.549897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-socket\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.549991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.549916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-certs\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.550076 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.550007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-envoy\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.550076 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.550039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-podinfo\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.550157 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.550111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-credential-socket\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.550157 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.550127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwscg\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-kube-api-access-vwscg\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.550232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.550173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-token\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.550232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.550212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-data\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-token\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-data\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istiod-ca-cert\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-socket\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-certs\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-envoy\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-podinfo\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-credential-socket\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwscg\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-kube-api-access-vwscg\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-data\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-socket\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.651870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.651854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-certs\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.652047 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.652025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-credential-socket\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.652213 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.652172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istiod-ca-cert\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.653820 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.653798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-envoy\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.654166 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.654146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-podinfo\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.663910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.663878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-token\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.665415 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.665394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwscg\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-kube-api-access-vwscg\") pod \"e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") " pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:53.764719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.764626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:53.806710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:53.806671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:54.080344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.080259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"]
Sep 14 10:39:54.083076 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:39:54.083048    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1088ec8_3c97_4a61_b983_baf4e9d12f2f.slice/crio-d495046a6d0d751e62483a74e21312aac5e0b8d705d88287e89d344da710586f WatchSource:0}: Error finding container d495046a6d0d751e62483a74e21312aac5e0b8d705d88287e89d344da710586f: Status 404 returned error can't find the container with id d495046a6d0d751e62483a74e21312aac5e0b8d705d88287e89d344da710586f
Sep 14 10:39:54.085172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.085142    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:39:54.085242 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.085204    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:39:54.085242 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.085230    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:39:54.237249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.237214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" event={"ID":"a1088ec8-3c97-4a61-b983-baf4e9d12f2f","Type":"ContainerStarted","Data":"3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb"}
Sep 14 10:39:54.237428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.237285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" event={"ID":"a1088ec8-3c97-4a61-b983-baf4e9d12f2f","Type":"ContainerStarted","Data":"d495046a6d0d751e62483a74e21312aac5e0b8d705d88287e89d344da710586f"}
Sep 14 10:39:54.262905 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.262855    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podStartSLOduration=1.26283944 podStartE2EDuration="1.26283944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:39:54.260822569 +0000 UTC m=+882.816010494" watchObservedRunningTime="2026-09-14 10:39:54.26283944 +0000 UTC m=+882.818027364"
Sep 14 10:39:54.807451 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:54.807423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:55.359617 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:55.359589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:55.359812 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:55.359660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:55.372038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:55.372016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:55.493281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:55.493206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:55.812779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:55.812750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:56.168098 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:56.168065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:56.248275 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:56.248204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:56.249016 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:56.248986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:39:56.259071 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:56.259052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:39:56.270890 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:56.270857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:57.193330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:57.193242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:58.002392 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:58.002357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:58.912336 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:58.912300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:59.594667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.594631    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"]
Sep 14 10:39:59.598053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.598031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.600377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.600301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 10:39:59.600377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.600337    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-cc7bcd-serving-cert\""
Sep 14 10:39:59.600611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.600465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bhlx\""
Sep 14 10:39:59.600847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.600831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 10:39:59.600908 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.600831    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:39:59.608327 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.608302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"]
Sep 14 10:39:59.629413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.629390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:59.713993 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.713964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6564e668-e257-49eb-91ea-803a875410d1-openshift-service-ca\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.714133 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.714012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-metrics-certs\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.714133 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.714109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghwbr\" (UniqueName: \"kubernetes.io/projected/6564e668-e257-49eb-91ea-803a875410d1-kube-api-access-ghwbr\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.714238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.714168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.731249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.731225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:39:59.814932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.814907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwbr\" (UniqueName: \"kubernetes.io/projected/6564e668-e257-49eb-91ea-803a875410d1-kube-api-access-ghwbr\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.815019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.814956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.815019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.814990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6564e668-e257-49eb-91ea-803a875410d1-openshift-service-ca\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.815137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.815022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-metrics-certs\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.815204 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:39:59.815136    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-serving-cert: secret "maas-api-e2e-explicit-cc7bcd-serving-cert" not found
Sep 14 10:39:59.815257 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:39:59.815218    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls podName:6564e668-e257-49eb-91ea-803a875410d1 nodeName:}" failed. No retries permitted until 2026-09-14 10:40:00.315197744 +0000 UTC m=+888.870385833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls") pod "maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b" (UID: "6564e668-e257-49eb-91ea-803a875410d1") : secret "maas-api-e2e-explicit-cc7bcd-serving-cert" not found
Sep 14 10:39:59.815628 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.815611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6564e668-e257-49eb-91ea-803a875410d1-openshift-service-ca\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.817627 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.817608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-metrics-certs\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:39:59.822932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:39:59.822914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwbr\" (UniqueName: \"kubernetes.io/projected/6564e668-e257-49eb-91ea-803a875410d1-kube-api-access-ghwbr\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:40:00.013981 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:00.013939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:00.115197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:00.115163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:00.213966 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:00.213925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:00.320108 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:00.320015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:40:00.322596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:00.322553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls\") pod \"maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:40:00.510687 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:00.510649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:40:00.778617 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:00.778586    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"]
Sep 14 10:40:00.781273 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:40:00.781245    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6564e668_e257_49eb_91ea_803a875410d1.slice/crio-7cbcca90f65bd4aefe1f6987bf95bbc7aba3078561e57ef77f2530c010272638 WatchSource:0}: Error finding container 7cbcca90f65bd4aefe1f6987bf95bbc7aba3078561e57ef77f2530c010272638: Status 404 returned error can't find the container with id 7cbcca90f65bd4aefe1f6987bf95bbc7aba3078561e57ef77f2530c010272638
Sep 14 10:40:01.276850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:01.276813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b" event={"ID":"6564e668-e257-49eb-91ea-803a875410d1","Type":"ContainerStarted","Data":"7cbcca90f65bd4aefe1f6987bf95bbc7aba3078561e57ef77f2530c010272638"}
Sep 14 10:40:02.283102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:02.283066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b" event={"ID":"6564e668-e257-49eb-91ea-803a875410d1","Type":"ContainerStarted","Data":"6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4"}
Sep 14 10:40:02.283428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:02.283163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:40:02.300342 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:02.300300    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b" podStartSLOduration=2.678526982 podStartE2EDuration="3.300286488s" totalImagesPullingTime="621.759506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:40:02.298998061 +0000 UTC m=+890.854185985" watchObservedRunningTime="2026-09-14 10:40:02.300286488 +0000 UTC m=+890.855474451"
Sep 14 10:40:03.668355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:03.668281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:40:06.802803 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:06.802757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:08.293245 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:08.293214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:40:12.791166 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:12.791124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:12.899048 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:12.899011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:12.993895 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:12.993860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:21.675934 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:21.675900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:22.355171 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:22.355133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:23.005283 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.005254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:23.067117 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.067040    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:40:23.405915 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.405883    2563 generic.go:356] "Generic (PLEG): container finished" podID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerID="3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb" exitCode=0
Sep 14 10:40:23.406092 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.405961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" event={"ID":"a1088ec8-3c97-4a61-b983-baf4e9d12f2f","Type":"ContainerDied","Data":"3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb"}
Sep 14 10:40:23.406408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.406392    2563 scope.go:117] "RemoveContainer" containerID="3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb"
Sep 14 10:40:23.407522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.407495    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:40:23.407645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.407601    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:40:23.407709 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.407653    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:40:23.519772 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:23.519733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:24.012791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:24.012747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:24.344049 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:24.343946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:24.415029 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:24.414984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" event={"ID":"a1088ec8-3c97-4a61-b983-baf4e9d12f2f","Type":"ContainerStarted","Data":"443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c"}
Sep 14 10:40:24.710737 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:24.710680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:24.807821 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:24.807784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:40:24.809044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:24.809015    2563 patch_prober.go:28] interesting pod/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 14 10:40:24.809179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:24.809093    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 14 10:40:25.394298 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:25.394267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:25.807344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:25.807259    2563 patch_prober.go:28] interesting pod/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 14 10:40:25.807344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:25.807318    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 14 10:40:26.248653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:26.248619    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:40:26.263837 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:26.263806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:26.807076 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:26.807041    2563 patch_prober.go:28] interesting pod/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 14 10:40:26.807437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:26.807102    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 14 10:40:27.807837 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:27.807801    2563 patch_prober.go:28] interesting pod/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 14 10:40:27.808198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:27.807851    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 14 10:40:28.807841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:28.807805    2563 patch_prober.go:28] interesting pod/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 14 10:40:28.808253 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:28.807869    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 14 10:40:29.811098 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:29.811071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:40:29.811935 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:29.811916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:40:30.304826 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:30.304781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:30.413183 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:30.413148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:30.525774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:30.525740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:30.672547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:30.672519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:31.064366 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:31.064280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:31.165082 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:31.165054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:31.305338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:31.305308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:31.401847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:31.401814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:31.497856 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:31.497822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:32.991732 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:32.991701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:39.230760 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:39.230726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:39.332990 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:39.332961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:39.911757 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:39.911728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:43.366283 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:43.366244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:40:46.000765 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:46.000733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:46.100449 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:46.100411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:46.239218 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:46.239195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:46.626339 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:46.626308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:46.773717 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:46.773680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:52.188345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:52.188300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:52.896997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:52.896955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:52.985959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:52.985915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:53.816607 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:53.816544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:54.831216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:54.831181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:55.486830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:55.486798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:56.367727 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:56.367691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:57.001314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.001282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"]
Sep 14 10:40:57.004009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.003992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.006250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.006224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-34925ca5-serving-cert\""
Sep 14 10:40:57.016686 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.016663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"]
Sep 14 10:40:57.044183 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.044155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:57.061008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.060982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-metrics-certs\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.061107 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.061058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcn8x\" (UniqueName: \"kubernetes.io/projected/034dd34a-0bbb-4cef-b023-b7c296f01d62-kube-api-access-tcn8x\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.061107 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.061101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034dd34a-0bbb-4cef-b023-b7c296f01d62-openshift-service-ca\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.061219 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.061122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.141354 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.141320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:57.161688 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.161653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-metrics-certs\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.161817 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.161758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcn8x\" (UniqueName: \"kubernetes.io/projected/034dd34a-0bbb-4cef-b023-b7c296f01d62-kube-api-access-tcn8x\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.161878 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.161814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034dd34a-0bbb-4cef-b023-b7c296f01d62-openshift-service-ca\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.161878 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.161847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.161984 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:40:57.161967    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-serving-cert: secret "maas-api-e2e-mt-34925ca5-serving-cert" not found
Sep 14 10:40:57.162065 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:40:57.162052    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls podName:034dd34a-0bbb-4cef-b023-b7c296f01d62 nodeName:}" failed. No retries permitted until 2026-09-14 10:40:57.662034152 +0000 UTC m=+946.217222076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls") pod "maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" (UID: "034dd34a-0bbb-4cef-b023-b7c296f01d62") : secret "maas-api-e2e-mt-34925ca5-serving-cert" not found
Sep 14 10:40:57.162435 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.162417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034dd34a-0bbb-4cef-b023-b7c296f01d62-openshift-service-ca\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.164187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.164166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-metrics-certs\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.170145 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.170114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcn8x\" (UniqueName: \"kubernetes.io/projected/034dd34a-0bbb-4cef-b023-b7c296f01d62-kube-api-access-tcn8x\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.589841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.589769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:57.665932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.665897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:57.666117 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:40:57.666070    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-serving-cert: secret "maas-api-e2e-mt-34925ca5-serving-cert" not found
Sep 14 10:40:57.666173 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:40:57.666159    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls podName:034dd34a-0bbb-4cef-b023-b7c296f01d62 nodeName:}" failed. No retries permitted until 2026-09-14 10:40:58.666137517 +0000 UTC m=+947.221325433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls") pod "maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" (UID: "034dd34a-0bbb-4cef-b023-b7c296f01d62") : secret "maas-api-e2e-mt-34925ca5-serving-cert" not found
Sep 14 10:40:57.691186 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.691157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:57.704540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.704511    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"]
Sep 14 10:40:57.707196 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.707179    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:57.709520 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.709503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-34925ca5\""
Sep 14 10:40:57.709644 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.709548    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-34925ca5-dockercfg-szbrp\""
Sep 14 10:40:57.718443 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.718420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"]
Sep 14 10:40:57.767449 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.767419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b2fce4-31e1-46d0-b983-e458a16c29c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:57.767663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.767533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j874r\" (UniqueName: \"kubernetes.io/projected/f8b2fce4-31e1-46d0-b983-e458a16c29c7-kube-api-access-j874r\") pod \"payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:57.868936 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.868838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b2fce4-31e1-46d0-b983-e458a16c29c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:57.868936 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.868925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j874r\" (UniqueName: \"kubernetes.io/projected/f8b2fce4-31e1-46d0-b983-e458a16c29c7-kube-api-access-j874r\") pod \"payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:57.869656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.869632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b2fce4-31e1-46d0-b983-e458a16c29c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:57.877353 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.877323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j874r\" (UniqueName: \"kubernetes.io/projected/f8b2fce4-31e1-46d0-b983-e458a16c29c7-kube-api-access-j874r\") pod \"payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:57.989352 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:57.989307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:40:58.018540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.018515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:58.288154 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.288124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"]
Sep 14 10:40:58.293245 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:40:58.293211    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b2fce4_31e1_46d0_b983_e458a16c29c7.slice/crio-86d86ae99a99b501d87a87bddf02d0ccdf4e62f76819b69f1ba2525998ea28d3 WatchSource:0}: Error finding container 86d86ae99a99b501d87a87bddf02d0ccdf4e62f76819b69f1ba2525998ea28d3: Status 404 returned error can't find the container with id 86d86ae99a99b501d87a87bddf02d0ccdf4e62f76819b69f1ba2525998ea28d3
Sep 14 10:40:58.599225 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.599117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8" event={"ID":"f8b2fce4-31e1-46d0-b983-e458a16c29c7","Type":"ContainerStarted","Data":"9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a"}
Sep 14 10:40:58.599637 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.599319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:40:58.599637 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.599392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8" event={"ID":"f8b2fce4-31e1-46d0-b983-e458a16c29c7","Type":"ContainerStarted","Data":"86d86ae99a99b501d87a87bddf02d0ccdf4e62f76819b69f1ba2525998ea28d3"}
Sep 14 10:40:58.614657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.614562    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8" podStartSLOduration=1.614541702 podStartE2EDuration="1.614541702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:40:58.613701314 +0000 UTC m=+947.168889240" watchObservedRunningTime="2026-09-14 10:40:58.614541702 +0000 UTC m=+947.169729630"
Sep 14 10:40:58.678008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.677971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:58.680583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.680547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls\") pod \"maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:58.815726 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:58.815695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:40:59.087891 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:59.087802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"]
Sep 14 10:40:59.090410 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:40:59.090387    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034dd34a_0bbb_4cef_b023_b7c296f01d62.slice/crio-73523f38a3d4c7e84af5aa834225569b96dd58641da7555695c2bc5dd4f02274 WatchSource:0}: Error finding container 73523f38a3d4c7e84af5aa834225569b96dd58641da7555695c2bc5dd4f02274: Status 404 returned error can't find the container with id 73523f38a3d4c7e84af5aa834225569b96dd58641da7555695c2bc5dd4f02274
Sep 14 10:40:59.605067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:40:59.605039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" event={"ID":"034dd34a-0bbb-4cef-b023-b7c296f01d62","Type":"ContainerStarted","Data":"73523f38a3d4c7e84af5aa834225569b96dd58641da7555695c2bc5dd4f02274"}
Sep 14 10:41:00.611972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:00.611940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" event={"ID":"034dd34a-0bbb-4cef-b023-b7c296f01d62","Type":"ContainerStarted","Data":"d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355"}
Sep 14 10:41:00.612383 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:00.612069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:41:00.631599 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:00.631539    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" podStartSLOduration=4.137338409 podStartE2EDuration="4.631525457s" totalImagesPullingTime="494.187048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:41:00.630688366 +0000 UTC m=+949.185876292" watchObservedRunningTime="2026-09-14 10:41:00.631525457 +0000 UTC m=+949.186713383"
Sep 14 10:41:04.151538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:04.151507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:06.622677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:06.622649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:41:09.607532 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:09.607498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:41:16.868074 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:16.868030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:41:18.567756 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.567724    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"]
Sep 14 10:41:18.571245 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.571220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.573634 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.573611    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1910e9\"/\"e2e-conflict-a-588c9108-kserve-self-signed-certs\""
Sep 14 10:41:18.583720 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.583698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"]
Sep 14 10:41:18.596956 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.596928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:18.659872 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.659839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-tmp-dir\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.660045 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.659892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-home\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.660045 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.659973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-dshm\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.660122 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.660046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-model-cache\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.660122 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.660088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e2680-7df1-4401-b559-7c1fdbf37a11-tls-certs\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.660189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.660126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srwm\" (UniqueName: \"kubernetes.io/projected/f08e2680-7df1-4401-b559-7c1fdbf37a11-kube-api-access-2srwm\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.761439 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-dshm\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.761636 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-model-cache\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.761636 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e2680-7df1-4401-b559-7c1fdbf37a11-tls-certs\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.761636 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srwm\" (UniqueName: \"kubernetes.io/projected/f08e2680-7df1-4401-b559-7c1fdbf37a11-kube-api-access-2srwm\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.761636 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-tmp-dir\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.761636 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-home\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.761961 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-model-cache\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.762027 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-home\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.762027 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.761989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-tmp-dir\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.763895 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.763864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-dshm\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.764137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.764118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e2680-7df1-4401-b559-7c1fdbf37a11-tls-certs\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.769736 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.769709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srwm\" (UniqueName: \"kubernetes.io/projected/f08e2680-7df1-4401-b559-7c1fdbf37a11-kube-api-access-2srwm\") pod \"e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") " pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:18.883453 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:18.883422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:19.164807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:19.164780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"]
Sep 14 10:41:19.168803 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:41:19.168763    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf08e2680_7df1_4401_b559_7c1fdbf37a11.slice/crio-58e471cf33d4cc13c724abf7686e2eceda46617cdb556d820036f9df86219dd9 WatchSource:0}: Error finding container 58e471cf33d4cc13c724abf7686e2eceda46617cdb556d820036f9df86219dd9: Status 404 returned error can't find the container with id 58e471cf33d4cc13c724abf7686e2eceda46617cdb556d820036f9df86219dd9
Sep 14 10:41:19.218363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:19.218331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:19.715129 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:19.715093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l" event={"ID":"f08e2680-7df1-4401-b559-7c1fdbf37a11","Type":"ContainerStarted","Data":"a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776"}
Sep 14 10:41:19.715129 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:19.715136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l" event={"ID":"f08e2680-7df1-4401-b559-7c1fdbf37a11","Type":"ContainerStarted","Data":"58e471cf33d4cc13c724abf7686e2eceda46617cdb556d820036f9df86219dd9"}
Sep 14 10:41:19.733917 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:19.733867    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l" podStartSLOduration=1.733850645 podStartE2EDuration="1.733850645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:41:19.733073466 +0000 UTC m=+968.288261391" watchObservedRunningTime="2026-09-14 10:41:19.733850645 +0000 UTC m=+968.289038599"
Sep 14 10:41:19.881114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:19.881083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:20.703921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:20.703888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:21.378840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:21.378800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:22.061398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:22.061363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:22.787236 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:22.787197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:22.930119 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:22.930080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:23.268729 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:23.268697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:23.614863 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:23.614780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:23.707831 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:23.707795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:23.809126 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:23.809092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:24.057695 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:24.057663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:24.629412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:24.629374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:25.396379 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:25.396346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:26.087757 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:26.087718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:26.872560 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:26.872512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:27.705881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:27.705840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:28.396706 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:28.396672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:28.883885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:28.883846    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:28.884074 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:28.883909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:28.895835 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:28.895807    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:29.212320 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:29.212222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:29.775524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:29.775496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:41:30.173754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:30.173725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:30.273544 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:30.273508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:31.496847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:31.496809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:32.196905 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:32.196873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:33.096325 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:33.096276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:33.786424 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:33.786356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:34.317281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:34.317246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:34.394652 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:34.394611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:34.702118 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:34.702084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:35.296522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.296483    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"]
Sep 14 10:41:35.296778 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.296752    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b" podUID="6564e668-e257-49eb-91ea-803a875410d1" containerName="maas-api" containerID="cri-o://6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4" gracePeriod=30
Sep 14 10:41:35.324306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.324277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:35.394521 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.394481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:35.494503 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.494467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:35.551179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.551126    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:41:35.573695 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.573662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:35.650302 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.650264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:35.729059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.729029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghwbr\" (UniqueName: \"kubernetes.io/projected/6564e668-e257-49eb-91ea-803a875410d1-kube-api-access-ghwbr\") pod \"6564e668-e257-49eb-91ea-803a875410d1\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") "
Sep 14 10:41:35.729249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.729068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-metrics-certs\") pod \"6564e668-e257-49eb-91ea-803a875410d1\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") "
Sep 14 10:41:35.729249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.729088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6564e668-e257-49eb-91ea-803a875410d1-openshift-service-ca\") pod \"6564e668-e257-49eb-91ea-803a875410d1\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") "
Sep 14 10:41:35.729249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.729135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls\") pod \"6564e668-e257-49eb-91ea-803a875410d1\" (UID: \"6564e668-e257-49eb-91ea-803a875410d1\") "
Sep 14 10:41:35.729631 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.729600    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6564e668-e257-49eb-91ea-803a875410d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6564e668-e257-49eb-91ea-803a875410d1" (UID: "6564e668-e257-49eb-91ea-803a875410d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:41:35.731359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.731333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6564e668-e257-49eb-91ea-803a875410d1" (UID: "6564e668-e257-49eb-91ea-803a875410d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:41:35.731441 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.731364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6564e668-e257-49eb-91ea-803a875410d1-kube-api-access-ghwbr" (OuterVolumeSpecName: "kube-api-access-ghwbr") pod "6564e668-e257-49eb-91ea-803a875410d1" (UID: "6564e668-e257-49eb-91ea-803a875410d1"). InnerVolumeSpecName "kube-api-access-ghwbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:41:35.731441 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.731385    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6564e668-e257-49eb-91ea-803a875410d1" (UID: "6564e668-e257-49eb-91ea-803a875410d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:41:35.795412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.795375    2563 generic.go:356] "Generic (PLEG): container finished" podID="6564e668-e257-49eb-91ea-803a875410d1" containerID="6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4" exitCode=0
Sep 14 10:41:35.795553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.795418    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"
Sep 14 10:41:35.795553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.795455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b" event={"ID":"6564e668-e257-49eb-91ea-803a875410d1","Type":"ContainerDied","Data":"6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4"}
Sep 14 10:41:35.795553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.795506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b" event={"ID":"6564e668-e257-49eb-91ea-803a875410d1","Type":"ContainerDied","Data":"7cbcca90f65bd4aefe1f6987bf95bbc7aba3078561e57ef77f2530c010272638"}
Sep 14 10:41:35.795553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.795528    2563 scope.go:117] "RemoveContainer" containerID="6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4"
Sep 14 10:41:35.805332 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.805315    2563 scope.go:117] "RemoveContainer" containerID="6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4"
Sep 14 10:41:35.805600 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:41:35.805557    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4\": container with ID starting with 6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4 not found: ID does not exist" containerID="6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4"
Sep 14 10:41:35.805663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.805610    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4"} err="failed to get container status \"6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4\": rpc error: code = NotFound desc = could not find container \"6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4\": container with ID starting with 6dc0a00e5674c6d41f29271921dd7634d261e282390309dfa80fd35376dcbcc4 not found: ID does not exist"
Sep 14 10:41:35.816651 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.816626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"]
Sep 14 10:41:35.821396 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.821378    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-cc7bcd-dfc59f95b-xxl8b"]
Sep 14 10:41:35.830758 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.830738    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:35.830848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.830761    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghwbr\" (UniqueName: \"kubernetes.io/projected/6564e668-e257-49eb-91ea-803a875410d1-kube-api-access-ghwbr\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:35.830848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.830774    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6564e668-e257-49eb-91ea-803a875410d1-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:35.830848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.830787    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6564e668-e257-49eb-91ea-803a875410d1-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:35.905037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.905007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:35.918658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:35.918628    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6564e668-e257-49eb-91ea-803a875410d1" path="/var/lib/kubelet/pods/6564e668-e257-49eb-91ea-803a875410d1/volumes"
Sep 14 10:41:36.415664 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:36.415628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:37.314010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:37.313976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:37.986301 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:37.986270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:38.684897 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:38.684860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:39.508488 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:39.508450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:40.262701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:40.262667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:40.898123 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:40.898082    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"]
Sep 14 10:41:40.898540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:40.898379    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" podUID="034dd34a-0bbb-4cef-b023-b7c296f01d62" containerName="maas-api" containerID="cri-o://d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355" gracePeriod=30
Sep 14 10:41:40.933049 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:40.933013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:40.966990 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:40.966961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"]
Sep 14 10:41:40.967210 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:40.967182    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8" podUID="f8b2fce4-31e1-46d0-b983-e458a16c29c7" containerName="payload-pre-processing" containerID="cri-o://9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a" gracePeriod=30
Sep 14 10:41:41.014269 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.014242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:41.117617 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.117584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:41.194177 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.194150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:41.233585 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.233541    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:41:41.236806 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.236783    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:41:41.281356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.281298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-metrics-certs\") pod \"034dd34a-0bbb-4cef-b023-b7c296f01d62\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") "
Sep 14 10:41:41.281557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.281377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls\") pod \"034dd34a-0bbb-4cef-b023-b7c296f01d62\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") "
Sep 14 10:41:41.281557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.281417    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcn8x\" (UniqueName: \"kubernetes.io/projected/034dd34a-0bbb-4cef-b023-b7c296f01d62-kube-api-access-tcn8x\") pod \"034dd34a-0bbb-4cef-b023-b7c296f01d62\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") "
Sep 14 10:41:41.281557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.281463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j874r\" (UniqueName: \"kubernetes.io/projected/f8b2fce4-31e1-46d0-b983-e458a16c29c7-kube-api-access-j874r\") pod \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") "
Sep 14 10:41:41.281557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.281488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034dd34a-0bbb-4cef-b023-b7c296f01d62-openshift-service-ca\") pod \"034dd34a-0bbb-4cef-b023-b7c296f01d62\" (UID: \"034dd34a-0bbb-4cef-b023-b7c296f01d62\") "
Sep 14 10:41:41.281557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.281519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b2fce4-31e1-46d0-b983-e458a16c29c7-plugins-config-volume\") pod \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\" (UID: \"f8b2fce4-31e1-46d0-b983-e458a16c29c7\") "
Sep 14 10:41:41.281987 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.281949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034dd34a-0bbb-4cef-b023-b7c296f01d62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "034dd34a-0bbb-4cef-b023-b7c296f01d62" (UID: "034dd34a-0bbb-4cef-b023-b7c296f01d62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:41:41.282385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.282356    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b2fce4-31e1-46d0-b983-e458a16c29c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8b2fce4-31e1-46d0-b983-e458a16c29c7" (UID: "f8b2fce4-31e1-46d0-b983-e458a16c29c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:41:41.283762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.283739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "034dd34a-0bbb-4cef-b023-b7c296f01d62" (UID: "034dd34a-0bbb-4cef-b023-b7c296f01d62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:41:41.283762 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.283744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034dd34a-0bbb-4cef-b023-b7c296f01d62-kube-api-access-tcn8x" (OuterVolumeSpecName: "kube-api-access-tcn8x") pod "034dd34a-0bbb-4cef-b023-b7c296f01d62" (UID: "034dd34a-0bbb-4cef-b023-b7c296f01d62"). InnerVolumeSpecName "kube-api-access-tcn8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:41:41.283937 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.283794    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "034dd34a-0bbb-4cef-b023-b7c296f01d62" (UID: "034dd34a-0bbb-4cef-b023-b7c296f01d62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:41:41.284116 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.284100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b2fce4-31e1-46d0-b983-e458a16c29c7-kube-api-access-j874r" (OuterVolumeSpecName: "kube-api-access-j874r") pod "f8b2fce4-31e1-46d0-b983-e458a16c29c7" (UID: "f8b2fce4-31e1-46d0-b983-e458a16c29c7"). InnerVolumeSpecName "kube-api-access-j874r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:41:41.382794 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.382759    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:41.382794 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.382791    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/034dd34a-0bbb-4cef-b023-b7c296f01d62-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:41.382794 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.382799    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcn8x\" (UniqueName: \"kubernetes.io/projected/034dd34a-0bbb-4cef-b023-b7c296f01d62-kube-api-access-tcn8x\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:41.383031 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.382808    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j874r\" (UniqueName: \"kubernetes.io/projected/f8b2fce4-31e1-46d0-b983-e458a16c29c7-kube-api-access-j874r\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:41.383031 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.382818    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/034dd34a-0bbb-4cef-b023-b7c296f01d62-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:41.383031 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.382827    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8b2fce4-31e1-46d0-b983-e458a16c29c7-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:41.499277 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.499198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:41.599456 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.599417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:41.827994 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.827901    2563 generic.go:356] "Generic (PLEG): container finished" podID="034dd34a-0bbb-4cef-b023-b7c296f01d62" containerID="d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355" exitCode=0
Sep 14 10:41:41.827994 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.827946    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"
Sep 14 10:41:41.827994 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.827985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" event={"ID":"034dd34a-0bbb-4cef-b023-b7c296f01d62","Type":"ContainerDied","Data":"d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355"}
Sep 14 10:41:41.828251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.828033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp" event={"ID":"034dd34a-0bbb-4cef-b023-b7c296f01d62","Type":"ContainerDied","Data":"73523f38a3d4c7e84af5aa834225569b96dd58641da7555695c2bc5dd4f02274"}
Sep 14 10:41:41.828251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.828056    2563 scope.go:117] "RemoveContainer" containerID="d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355"
Sep 14 10:41:41.829469 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.829450    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8b2fce4-31e1-46d0-b983-e458a16c29c7" containerID="9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a" exitCode=0
Sep 14 10:41:41.829602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.829495    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"
Sep 14 10:41:41.829602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.829541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8" event={"ID":"f8b2fce4-31e1-46d0-b983-e458a16c29c7","Type":"ContainerDied","Data":"9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a"}
Sep 14 10:41:41.829724 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.829608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8" event={"ID":"f8b2fce4-31e1-46d0-b983-e458a16c29c7","Type":"ContainerDied","Data":"86d86ae99a99b501d87a87bddf02d0ccdf4e62f76819b69f1ba2525998ea28d3"}
Sep 14 10:41:41.837890 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.837869    2563 scope.go:117] "RemoveContainer" containerID="d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355"
Sep 14 10:41:41.838224 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:41:41.838206    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355\": container with ID starting with d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355 not found: ID does not exist" containerID="d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355"
Sep 14 10:41:41.838316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.838229    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355"} err="failed to get container status \"d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355\": rpc error: code = NotFound desc = could not find container \"d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355\": container with ID starting with d66247fd04ba420b4de456f3c7f8b6770c4c782a11c373a8ee2f25f79ec8e355 not found: ID does not exist"
Sep 14 10:41:41.838316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.838247    2563 scope.go:117] "RemoveContainer" containerID="9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a"
Sep 14 10:41:41.846687 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.846664    2563 scope.go:117] "RemoveContainer" containerID="9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a"
Sep 14 10:41:41.847656 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:41:41.847386    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a\": container with ID starting with 9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a not found: ID does not exist" containerID="9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a"
Sep 14 10:41:41.847656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.847425    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a"} err="failed to get container status \"9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a\": rpc error: code = NotFound desc = could not find container \"9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a\": container with ID starting with 9f193d4780c58bdda8741887a75900e38073890cb7b260b48e847a9a4bc12c9a not found: ID does not exist"
Sep 14 10:41:41.851250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.851226    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"]
Sep 14 10:41:41.853469 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.853451    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34925ca5-89c8bd76c-rp4jp"]
Sep 14 10:41:41.864166 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.864145    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"]
Sep 14 10:41:41.869121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.869101    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34925ca5-5dd455bcc8-mhhd8"]
Sep 14 10:41:41.925738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.925714    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="034dd34a-0bbb-4cef-b023-b7c296f01d62" path="/var/lib/kubelet/pods/034dd34a-0bbb-4cef-b023-b7c296f01d62/volumes"
Sep 14 10:41:41.926085 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:41.926072    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b2fce4-31e1-46d0-b983-e458a16c29c7" path="/var/lib/kubelet/pods/f8b2fce4-31e1-46d0-b983-e458a16c29c7/volumes"
Sep 14 10:41:42.194259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:42.194227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:42.749262 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:42.749228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:42.842068 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:42.842040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:42.941192 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:42.941144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:42.992180 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:42.992149    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"]
Sep 14 10:41:42.992376 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:42.992354    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy" containerID="cri-o://443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c" gracePeriod=30
Sep 14 10:41:43.020161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.020094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:43.468745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.468709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:43.752628 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.752536    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"]
Sep 14 10:41:43.752959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.752944    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="034dd34a-0bbb-4cef-b023-b7c296f01d62" containerName="maas-api"
Sep 14 10:41:43.753007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.752960    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="034dd34a-0bbb-4cef-b023-b7c296f01d62" containerName="maas-api"
Sep 14 10:41:43.753007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.752969    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b2fce4-31e1-46d0-b983-e458a16c29c7" containerName="payload-pre-processing"
Sep 14 10:41:43.753007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.752974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b2fce4-31e1-46d0-b983-e458a16c29c7" containerName="payload-pre-processing"
Sep 14 10:41:43.753007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.752982    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6564e668-e257-49eb-91ea-803a875410d1" containerName="maas-api"
Sep 14 10:41:43.753007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.752988    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6564e668-e257-49eb-91ea-803a875410d1" containerName="maas-api"
Sep 14 10:41:43.753172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.753058    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b2fce4-31e1-46d0-b983-e458a16c29c7" containerName="payload-pre-processing"
Sep 14 10:41:43.753172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.753068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6564e668-e257-49eb-91ea-803a875410d1" containerName="maas-api"
Sep 14 10:41:43.753172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.753077    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="034dd34a-0bbb-4cef-b023-b7c296f01d62" containerName="maas-api"
Sep 14 10:41:43.757722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.757702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.759884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.759865    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-f68032-openshift-default-dockercfg-db4ts\""
Sep 14 10:41:43.770188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.770165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"]
Sep 14 10:41:43.783915 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.783886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:43.806062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-data\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806178 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c23ec3-adee-41bd-aa95-540e3b426d99-istiod-ca-cert\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806178 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-socket\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806289 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-credential-socket\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806289 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-certs\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806289 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-token\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-envoy\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljcj\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-kube-api-access-7ljcj\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.806437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.806404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-podinfo\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.890165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.890128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:43.907843 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.907813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-envoy\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.907959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.907849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljcj\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-kube-api-access-7ljcj\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.907959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.907897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-podinfo\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908073 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.907971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-data\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908073 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c23ec3-adee-41bd-aa95-540e3b426d99-istiod-ca-cert\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-socket\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-credential-socket\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-certs\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-token\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908440 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-data\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908545 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-socket\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908625 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-certs\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-credential-socket\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.908985 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.908966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c23ec3-adee-41bd-aa95-540e3b426d99-istiod-ca-cert\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.910815 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.910798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-envoy\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.911126 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.911109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-podinfo\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.916190 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.916167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljcj\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-kube-api-access-7ljcj\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:43.916281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:43.916225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-token\") pod \"e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") " pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:44.067551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.067469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:44.128471 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.128436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:44.351190 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.351164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"]
Sep 14 10:41:44.353714 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:41:44.353685    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c23ec3_adee_41bd_aa95_540e3b426d99.slice/crio-56c432d6d06fece5823a51589342aee7109b73e26e623f41dab9f65fe2c304d8 WatchSource:0}: Error finding container 56c432d6d06fece5823a51589342aee7109b73e26e623f41dab9f65fe2c304d8: Status 404 returned error can't find the container with id 56c432d6d06fece5823a51589342aee7109b73e26e623f41dab9f65fe2c304d8
Sep 14 10:41:44.356116 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.356070    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:41:44.356221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.356166    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:41:44.356221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.356214    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:41:44.605620 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.605507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:44.813780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.813743    2563 patch_prober.go:28] interesting pod/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 14 10:41:44.813932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.813810    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 14 10:41:44.848341 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.848298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm" event={"ID":"89c23ec3-adee-41bd-aa95-540e3b426d99","Type":"ContainerStarted","Data":"e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b"}
Sep 14 10:41:44.848489 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.848352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm" event={"ID":"89c23ec3-adee-41bd-aa95-540e3b426d99","Type":"ContainerStarted","Data":"56c432d6d06fece5823a51589342aee7109b73e26e623f41dab9f65fe2c304d8"}
Sep 14 10:41:44.868535 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:44.868447    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm" podStartSLOduration=1.8684336849999998 podStartE2EDuration="1.868433685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:41:44.866615886 +0000 UTC m=+993.421803804" watchObservedRunningTime="2026-09-14 10:41:44.868433685 +0000 UTC m=+993.423621610"
Sep 14 10:41:45.067609 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:45.067552    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:45.073813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:45.073788    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:45.235064 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:45.235032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:45.366279 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:45.366245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:45.464051 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:45.464021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:45.854345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:45.854299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:45.861179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:45.861153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:41:46.865728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:46.865688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:47.680315 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:47.680281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:47.750963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:47.750924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:47.952791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:47.952707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:48.236292 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.236264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:41:48.275786 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.275746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:48.346855 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.346823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:48.352940 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.352906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwscg\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-kube-api-access-vwscg\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.352986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-token\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353043    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-envoy\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-podinfo\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353133    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-socket\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-certs\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-data\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istiod-ca-cert\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-credential-socket\") pod \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\" (UID: \"a1088ec8-3c97-4a61-b983-baf4e9d12f2f\") "
Sep 14 10:41:48.353473 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353447    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:41:48.353527 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:41:48.353716 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:41:48.353805 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:41:48.353805 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353761    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.353805 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353780    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.353805 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353791    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.354000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.353978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:41:48.355432 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.355406    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:41:48.355542 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.355446    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:41:48.355542 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.355451    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-kube-api-access-vwscg" (OuterVolumeSpecName: "kube-api-access-vwscg") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "kube-api-access-vwscg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:41:48.355925 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.355905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1088ec8-3c97-4a61-b983-baf4e9d12f2f" (UID: "a1088ec8-3c97-4a61-b983-baf4e9d12f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:41:48.454768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.454739    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.454768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.454767    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.454768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.454775    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.455000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.454785    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.455000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.454793    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwscg\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-kube-api-access-vwscg\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.455000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.454803    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1088ec8-3c97-4a61-b983-baf4e9d12f2f-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:41:48.483404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.483331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:48.872437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.872352    2563 generic.go:356] "Generic (PLEG): container finished" podID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerID="443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c" exitCode=0
Sep 14 10:41:48.872437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.872405    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"
Sep 14 10:41:48.872650 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.872437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" event={"ID":"a1088ec8-3c97-4a61-b983-baf4e9d12f2f","Type":"ContainerDied","Data":"443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c"}
Sep 14 10:41:48.872650 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.872498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg" event={"ID":"a1088ec8-3c97-4a61-b983-baf4e9d12f2f","Type":"ContainerDied","Data":"d495046a6d0d751e62483a74e21312aac5e0b8d705d88287e89d344da710586f"}
Sep 14 10:41:48.872650 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.872526    2563 scope.go:117] "RemoveContainer" containerID="443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c"
Sep 14 10:41:48.882288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.882270    2563 scope.go:117] "RemoveContainer" containerID="3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb"
Sep 14 10:41:48.890449 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.890431    2563 scope.go:117] "RemoveContainer" containerID="443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c"
Sep 14 10:41:48.890737 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:41:48.890718    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c\": container with ID starting with 443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c not found: ID does not exist" containerID="443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c"
Sep 14 10:41:48.890806 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.890746    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c"} err="failed to get container status \"443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c\": rpc error: code = NotFound desc = could not find container \"443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c\": container with ID starting with 443165a44f9107527bbb3ea5d23d1c18f251f515f65c43335e041ca147ecf94c not found: ID does not exist"
Sep 14 10:41:48.890806 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.890763    2563 scope.go:117] "RemoveContainer" containerID="3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb"
Sep 14 10:41:48.891048 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:41:48.891026    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb\": container with ID starting with 3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb not found: ID does not exist" containerID="3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb"
Sep 14 10:41:48.891121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.891057    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb"} err="failed to get container status \"3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb\": rpc error: code = NotFound desc = could not find container \"3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb\": container with ID starting with 3ff7558c1381cd266e60e1cb6390e3ff4a36c2b04abb4f1daf3bd39b116d89fb not found: ID does not exist"
Sep 14 10:41:48.894883 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.894860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"]
Sep 14 10:41:48.899897 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:48.899878    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-cc7bcd-openshift-default-5bfbbbfdc6-vdgbg"]
Sep 14 10:41:49.918891 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:49.918857    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" path="/var/lib/kubelet/pods/a1088ec8-3c97-4a61-b983-baf4e9d12f2f/volumes"
Sep 14 10:41:51.896371 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:51.896339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:54.257528 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:54.257493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:54.347508 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:54.347480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:54.443195 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:54.443163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:54.648237 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:54.648202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:41:56.167106 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:41:56.167069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:42:00.560974 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:00.560941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:00.671610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:00.671551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:01.483670 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:01.483640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:01.582398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:01.582361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:01.817909 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:01.817803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:02.113374 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:02.113227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:02.949319 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:02.949280    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"]
Sep 14 10:42:02.973099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:02.973058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:04.094213 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:04.094179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:04.792019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:04.791987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:04.904942 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:04.904909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:04.993251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:04.993214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:05.796821 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:05.796793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:06.691445 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:06.691415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:07.491102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:07.491073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:08.374702 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.374669    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"]
Sep 14 10:42:08.375154 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.375131    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy"
Sep 14 10:42:08.375154 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.375151    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy"
Sep 14 10:42:08.375347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.375165    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy"
Sep 14 10:42:08.375347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.375171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy"
Sep 14 10:42:08.375347 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.375299    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy"
Sep 14 10:42:08.378216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.378190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.381398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.381379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bhlx\""
Sep 14 10:42:08.381540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.381522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 10:42:08.381632 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.381379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 10:42:08.381632 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.381421    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 10:42:08.381748 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.381421    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-3503d7-serving-cert\""
Sep 14 10:42:08.391963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.391936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"]
Sep 14 10:42:08.526793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.526757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:08.550814 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.550785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.550939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.550834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.550939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.550932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqht\" (UniqueName: \"kubernetes.io/projected/48fceceb-9b6d-4c90-b790-ee257746e4ca-kube-api-access-2wqht\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.551020 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.550952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fceceb-9b6d-4c90-b790-ee257746e4ca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.627813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.627752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:08.651916 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.651887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fceceb-9b6d-4c90-b790-ee257746e4ca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.652000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.651957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.652000 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.651986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.652078 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.652025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqht\" (UniqueName: \"kubernetes.io/projected/48fceceb-9b6d-4c90-b790-ee257746e4ca-kube-api-access-2wqht\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.652132 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:42:08.652114    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-serving-cert: secret "maas-api-e2e-shared-a-w2-3503d7-serving-cert" not found
Sep 14 10:42:08.652205 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:42:08.652194    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls podName:48fceceb-9b6d-4c90-b790-ee257746e4ca nodeName:}" failed. No retries permitted until 2026-09-14 10:42:09.152175026 +0000 UTC m=+1017.707362930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls") pod "maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp" (UID: "48fceceb-9b6d-4c90-b790-ee257746e4ca") : secret "maas-api-e2e-shared-a-w2-3503d7-serving-cert" not found
Sep 14 10:42:08.652500 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.652481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fceceb-9b6d-4c90-b790-ee257746e4ca-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.654465 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.654446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.662884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.662864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqht\" (UniqueName: \"kubernetes.io/projected/48fceceb-9b6d-4c90-b790-ee257746e4ca-kube-api-access-2wqht\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:08.728326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.728294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:08.909662 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:08.909584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:09.012141 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:09.012111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:09.156807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:09.156772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:09.159282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:09.159261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:09.291232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:09.291146    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:09.563386 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:09.563334    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"]
Sep 14 10:42:09.565693 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:42:09.565666    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fceceb_9b6d_4c90_b790_ee257746e4ca.slice/crio-e86fe54c6e9f3474c2f3894a4d51fd09982a4c55d39dd3fc20bce15d751132ab WatchSource:0}: Error finding container e86fe54c6e9f3474c2f3894a4d51fd09982a4c55d39dd3fc20bce15d751132ab: Status 404 returned error can't find the container with id e86fe54c6e9f3474c2f3894a4d51fd09982a4c55d39dd3fc20bce15d751132ab
Sep 14 10:42:09.567008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:09.566990    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:42:09.980533 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:09.980500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp" event={"ID":"48fceceb-9b6d-4c90-b790-ee257746e4ca","Type":"ContainerStarted","Data":"e86fe54c6e9f3474c2f3894a4d51fd09982a4c55d39dd3fc20bce15d751132ab"}
Sep 14 10:42:10.987448 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:10.987414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp" event={"ID":"48fceceb-9b6d-4c90-b790-ee257746e4ca","Type":"ContainerStarted","Data":"3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51"}
Sep 14 10:42:10.987828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:10.987475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:11.012721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:11.012676    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp" podStartSLOduration=2.445945384 podStartE2EDuration="3.012661538s" totalImagesPullingTime="566.716154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:42:11.010663213 +0000 UTC m=+1019.565851140" watchObservedRunningTime="2026-09-14 10:42:11.012661538 +0000 UTC m=+1019.567849464"
Sep 14 10:42:16.997909 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:16.997883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:42:17.968288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:17.968253    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6" podUID="ad047ad2-0dfe-40b1-a155-33a6f609bd92" containerName="main" containerID="cri-o://26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925" gracePeriod=45
Sep 14 10:42:18.209555 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.209534    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:42:18.233083 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-model-cache\") pod \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") "
Sep 14 10:42:18.233083 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-home\") pod \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") "
Sep 14 10:42:18.233261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tls-certs\") pod \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") "
Sep 14 10:42:18.233261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-dshm\") pod \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") "
Sep 14 10:42:18.233261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tmp-dir\") pod \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") "
Sep 14 10:42:18.233261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/ad047ad2-0dfe-40b1-a155-33a6f609bd92-kube-api-access-q2dzr\") pod \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\" (UID: \"ad047ad2-0dfe-40b1-a155-33a6f609bd92\") "
Sep 14 10:42:18.233479 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-home" (OuterVolumeSpecName: "home") pod "ad047ad2-0dfe-40b1-a155-33a6f609bd92" (UID: "ad047ad2-0dfe-40b1-a155-33a6f609bd92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:42:18.233479 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad047ad2-0dfe-40b1-a155-33a6f609bd92" (UID: "ad047ad2-0dfe-40b1-a155-33a6f609bd92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:42:18.233599 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad047ad2-0dfe-40b1-a155-33a6f609bd92" (UID: "ad047ad2-0dfe-40b1-a155-33a6f609bd92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:42:18.233667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233618    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:42:18.233667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233636    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:42:18.233667 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.233649    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:42:18.235492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.235469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad047ad2-0dfe-40b1-a155-33a6f609bd92" (UID: "ad047ad2-0dfe-40b1-a155-33a6f609bd92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:42:18.235871 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.235842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-dshm" (OuterVolumeSpecName: "dshm") pod "ad047ad2-0dfe-40b1-a155-33a6f609bd92" (UID: "ad047ad2-0dfe-40b1-a155-33a6f609bd92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:42:18.235964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.235941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad047ad2-0dfe-40b1-a155-33a6f609bd92-kube-api-access-q2dzr" (OuterVolumeSpecName: "kube-api-access-q2dzr") pod "ad047ad2-0dfe-40b1-a155-33a6f609bd92" (UID: "ad047ad2-0dfe-40b1-a155-33a6f609bd92"). InnerVolumeSpecName "kube-api-access-q2dzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:42:18.334197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.334174    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad047ad2-0dfe-40b1-a155-33a6f609bd92-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:42:18.334197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.334196    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad047ad2-0dfe-40b1-a155-33a6f609bd92-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:42:18.334320 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:18.334205    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2dzr\" (UniqueName: \"kubernetes.io/projected/ad047ad2-0dfe-40b1-a155-33a6f609bd92-kube-api-access-q2dzr\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:42:19.031010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.030978    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad047ad2-0dfe-40b1-a155-33a6f609bd92" containerID="26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925" exitCode=0
Sep 14 10:42:19.031168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.031016    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"
Sep 14 10:42:19.031168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.031058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6" event={"ID":"ad047ad2-0dfe-40b1-a155-33a6f609bd92","Type":"ContainerDied","Data":"26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925"}
Sep 14 10:42:19.031168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.031095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6" event={"ID":"ad047ad2-0dfe-40b1-a155-33a6f609bd92","Type":"ContainerDied","Data":"ee1b32cd4f224228bd6e7da010394037e1827350f60e8f2b1ca3ac1d032b6692"}
Sep 14 10:42:19.032036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.031113    2563 scope.go:117] "RemoveContainer" containerID="26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925"
Sep 14 10:42:19.041251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.041236    2563 scope.go:117] "RemoveContainer" containerID="26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925"
Sep 14 10:42:19.041511 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:42:19.041494    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925\": container with ID starting with 26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925 not found: ID does not exist" containerID="26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925"
Sep 14 10:42:19.041552 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.041522    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925"} err="failed to get container status \"26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925\": rpc error: code = NotFound desc = could not find container \"26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925\": container with ID starting with 26019c265ea50c09bf640e37686f6f695a3c943da297fbfd16a2cf88c3f9b925 not found: ID does not exist"
Sep 14 10:42:19.051939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.051917    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"]
Sep 14 10:42:19.056504 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.056484    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d36f67/test-model-d36f67-kserve-77477589f7-mjbp6"]
Sep 14 10:42:19.918849 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:19.918817    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad047ad2-0dfe-40b1-a155-33a6f609bd92" path="/var/lib/kubelet/pods/ad047ad2-0dfe-40b1-a155-33a6f609bd92/volumes"
Sep 14 10:42:21.263938 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:21.263899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:42:30.596981 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:30.596949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:31.292461 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:31.292428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:32.038320 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:32.038285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:32.914995 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:32.914956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:33.627593 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:33.627536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:34.462266 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:34.462225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:34.961323 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:34.961293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:35.170491 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.170458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"]
Sep 14 10:42:35.171055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.171033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad047ad2-0dfe-40b1-a155-33a6f609bd92" containerName="main"
Sep 14 10:42:35.171055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.171055    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad047ad2-0dfe-40b1-a155-33a6f609bd92" containerName="main"
Sep 14 10:42:35.171207 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.171141    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad047ad2-0dfe-40b1-a155-33a6f609bd92" containerName="main"
Sep 14 10:42:35.171207 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.171152    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1088ec8-3c97-4a61-b983-baf4e9d12f2f" containerName="istio-proxy"
Sep 14 10:42:35.175665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.175643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.178113 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.178090    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f68032\"/\"kube-root-ca.crt\""
Sep 14 10:42:35.178221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.178090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f68032\"/\"noten-model-f68032-kserve-self-signed-certs\""
Sep 14 10:42:35.178221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.178125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f68032\"/\"default-dockercfg-b2pp5\""
Sep 14 10:42:35.178221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.178090    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f68032\"/\"openshift-service-ca.crt\""
Sep 14 10:42:35.183250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.183008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"]
Sep 14 10:42:35.193284 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.193254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:35.281692 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.281612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tmp-dir\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.281692 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.281654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hgvq\" (UniqueName: \"kubernetes.io/projected/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-kube-api-access-9hgvq\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.281860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.281711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-home\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.281860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.281757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-dshm\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.281860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.281802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tls-certs\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.281860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.281844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-model-cache\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.329919 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.329883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:35.382670 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.382628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-dshm\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.382870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.382682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tls-certs\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.382870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.382719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-model-cache\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.382870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.382800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tmp-dir\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.383035 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.382938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hgvq\" (UniqueName: \"kubernetes.io/projected/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-kube-api-access-9hgvq\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.383035 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.383022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-home\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.383138 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.383108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tmp-dir\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.383191 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.383150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-model-cache\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.383306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.383291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-home\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.385144 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.385117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-dshm\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.385429 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.385410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tls-certs\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.390693 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.390673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hgvq\" (UniqueName: \"kubernetes.io/projected/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-kube-api-access-9hgvq\") pod \"noten-model-f68032-kserve-54f988b699-r7fnp\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") " pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.486174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.486131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:35.600505 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.600475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:35.758657 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:35.758550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"]
Sep 14 10:42:35.761413 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:42:35.761384    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23bc6cb2_21dc_4968_9fc0_0825e7244a4e.slice/crio-1c5acae1ba812d97697845239519852ed5843c4a6b629f9c0aae16d8a935588f WatchSource:0}: Error finding container 1c5acae1ba812d97697845239519852ed5843c4a6b629f9c0aae16d8a935588f: Status 404 returned error can't find the container with id 1c5acae1ba812d97697845239519852ed5843c4a6b629f9c0aae16d8a935588f
Sep 14 10:42:36.128984 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:36.128948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp" event={"ID":"23bc6cb2-21dc-4968-9fc0-0825e7244a4e","Type":"ContainerStarted","Data":"004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7"}
Sep 14 10:42:36.129400 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:36.128998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp" event={"ID":"23bc6cb2-21dc-4968-9fc0-0825e7244a4e","Type":"ContainerStarted","Data":"1c5acae1ba812d97697845239519852ed5843c4a6b629f9c0aae16d8a935588f"}
Sep 14 10:42:36.138553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:36.138531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:36.147935 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:36.147878    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp" podStartSLOduration=1.147861918 podStartE2EDuration="1.147861918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:42:36.145842509 +0000 UTC m=+1044.701030435" watchObservedRunningTime="2026-09-14 10:42:36.147861918 +0000 UTC m=+1044.703049844"
Sep 14 10:42:38.380142 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:38.380112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:39.128965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:39.128931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:39.215642 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:39.215607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:40.131177 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:40.131137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:40.894336 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:40.894300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:41.460070 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:41.460037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:41.573811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:41.573775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:41.661897 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:41.661868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:41.734015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:41.733928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:41.981434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:41.981398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:42.089995 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:42.089919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:42.706611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:42.706549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:42.825755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:42.825720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:42.980409 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:42.980316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:43.351921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:43.351835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:43.452891 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:43.452863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:43.840349 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:43.840301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:44.693820 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:44.693790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:45.468528 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:45.468495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:45.486659 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:45.486632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:45.486759 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:45.486674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:45.498834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:45.498814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:46.187614 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:46.187560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:46.191544 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:46.191528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:42:47.012492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:47.012468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:48.002528 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:48.002493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:48.832681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:48.832653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:48.938959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:48.938927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:49.110147 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:49.110056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:49.758367 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:49.758331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:50.598081 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:50.598042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:50.882299 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:50.882266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:42:51.324693 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:51.324585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:51.551721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:51.551687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:52.166848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:52.166819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:52.987227 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:52.987180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:53.765314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:53.765272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:54.609253 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:54.609227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:55.288750 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:55.288714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:55.992341 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:55.992309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:56.801793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:56.801759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:57.488393 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:57.488356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:58.263051 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:58.263005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:58.920524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:58.920478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:59.030621 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:59.030565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:59.641043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:59.641015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:59.711560 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:59.711531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:59.773757 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:59.773720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:42:59.838825 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:42:59.838779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:00.189247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:00.189203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:00.293878 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:00.293845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:01.092943 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:01.092909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:01.786849 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:01.786793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:02.627511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:02.627474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:03.494335 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:03.494290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:04.186175 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:04.186137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:06.376450 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:06.376418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:06.476549 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:06.476517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:06.641055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:06.640972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:07.183417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:07.183370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:07.496046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:07.495943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:08.167881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:08.167843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:08.404204 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:08.404167    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"]
Sep 14 10:43:08.404414 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:08.404388    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm" podUID="89c23ec3-adee-41bd-aa95-540e3b426d99" containerName="istio-proxy" containerID="cri-o://e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b" gracePeriod=30
Sep 14 10:43:08.429380 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:08.429254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:08.618168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:08.618132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:08.718547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:08.718461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:08.777703 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:08.777672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:09.088109 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:09.088028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:09.206970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:09.206937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:10.201868 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:10.201834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:10.390326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:10.390280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:10.490477 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:10.490400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:11.036195 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:11.036168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:11.981154 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:11.981126    2563 scope.go:117] "RemoveContainer" containerID="9fd672305faa41c7509a679ca7fcd5a79115e362cc460bcc7d82c193a1a05827"
Sep 14 10:43:11.989625 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:11.989604    2563 scope.go:117] "RemoveContainer" containerID="1d868b31112f1632bad21308c91965b880251a7e7dfc2a3c859f586da9aaab76"
Sep 14 10:43:12.102669 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:12.102642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:12.350640 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:12.350543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:12.427936 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:12.427909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:13.091435 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.091405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:13.655034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.655012    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:43:13.722699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.722626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"]
Sep 14 10:43:13.763920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.763894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ljcj\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-kube-api-access-7ljcj\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.763971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-podinfo\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-credential-socket\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764191    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c23ec3-adee-41bd-aa95-540e3b426d99-istiod-ca-cert\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764234    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-data\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:13.764350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764286    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-certs\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-token\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-socket\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-envoy\") pod \"89c23ec3-adee-41bd-aa95-540e3b426d99\" (UID: \"89c23ec3-adee-41bd-aa95-540e3b426d99\") "
Sep 14 10:43:13.764752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764477    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:13.764752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c23ec3-adee-41bd-aa95-540e3b426d99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:43:13.764752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764709    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-data" (OuterVolumeSpecName: "istio-data") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:13.764752 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.764735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:13.765315 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.765296    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.765377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.765323    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c23ec3-adee-41bd-aa95-540e3b426d99-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.765377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.765346    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.765377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.765360    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.765377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.765372    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.766203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.766182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:43:13.766439 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.766415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-token" (OuterVolumeSpecName: "istio-token") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:43:13.766560 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.766523    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-kube-api-access-7ljcj" (OuterVolumeSpecName: "kube-api-access-7ljcj") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "kube-api-access-7ljcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:43:13.766765 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.766752    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "89c23ec3-adee-41bd-aa95-540e3b426d99" (UID: "89c23ec3-adee-41bd-aa95-540e3b426d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:13.801165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.801133    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"]
Sep 14 10:43:13.801550 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.801519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:13.801791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.801773    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c23ec3-adee-41bd-aa95-540e3b426d99" containerName="istio-proxy"
Sep 14 10:43:13.801866 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.801794    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c23ec3-adee-41bd-aa95-540e3b426d99" containerName="istio-proxy"
Sep 14 10:43:13.801951 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.801939    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c23ec3-adee-41bd-aa95-540e3b426d99" containerName="istio-proxy"
Sep 14 10:43:13.805447 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.805425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.807377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.807358    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1cbcd0-serving-cert\""
Sep 14 10:43:13.814249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.814189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"]
Sep 14 10:43:13.865958 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.865930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.866084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.865986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44znj\" (UniqueName: \"kubernetes.io/projected/b8161d19-36ea-4edc-a29c-eaa7890d5238-kube-api-access-44znj\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.866084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.866009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.866084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.866057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8161d19-36ea-4edc-a29c-eaa7890d5238-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.866193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.866095    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.866193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.866133    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.866193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.866142    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c23ec3-adee-41bd-aa95-540e3b426d99-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.866193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.866150    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljcj\" (UniqueName: \"kubernetes.io/projected/89c23ec3-adee-41bd-aa95-540e3b426d99-kube-api-access-7ljcj\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:13.882315 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.882290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:13.967400 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.967367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.967519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.967480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44znj\" (UniqueName: \"kubernetes.io/projected/b8161d19-36ea-4edc-a29c-eaa7890d5238-kube-api-access-44znj\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.967519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.967513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.967671 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.967615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8161d19-36ea-4edc-a29c-eaa7890d5238-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.967730 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:43:13.967672    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-serving-cert: secret "maas-api-e2e-shared-a-w3-1cbcd0-serving-cert" not found
Sep 14 10:43:13.967782 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:43:13.967748    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls podName:b8161d19-36ea-4edc-a29c-eaa7890d5238 nodeName:}" failed. No retries permitted until 2026-09-14 10:43:14.467726919 +0000 UTC m=+1083.022914824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx" (UID: "b8161d19-36ea-4edc-a29c-eaa7890d5238") : secret "maas-api-e2e-shared-a-w3-1cbcd0-serving-cert" not found
Sep 14 10:43:13.968203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.968180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8161d19-36ea-4edc-a29c-eaa7890d5238-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.969881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.969862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:13.975676 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:13.975616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44znj\" (UniqueName: \"kubernetes.io/projected/b8161d19-36ea-4edc-a29c-eaa7890d5238-kube-api-access-44znj\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:14.044393 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.044368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:14.332726 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.332653    2563 generic.go:356] "Generic (PLEG): container finished" podID="89c23ec3-adee-41bd-aa95-540e3b426d99" containerID="e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b" exitCode=0
Sep 14 10:43:14.332726 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.332694    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"
Sep 14 10:43:14.333148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.332735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm" event={"ID":"89c23ec3-adee-41bd-aa95-540e3b426d99","Type":"ContainerDied","Data":"e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b"}
Sep 14 10:43:14.333148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.332784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm" event={"ID":"89c23ec3-adee-41bd-aa95-540e3b426d99","Type":"ContainerDied","Data":"56c432d6d06fece5823a51589342aee7109b73e26e623f41dab9f65fe2c304d8"}
Sep 14 10:43:14.333148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.332807    2563 scope.go:117] "RemoveContainer" containerID="e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b"
Sep 14 10:43:14.341471 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.341452    2563 scope.go:117] "RemoveContainer" containerID="e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b"
Sep 14 10:43:14.341769 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:43:14.341745    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b\": container with ID starting with e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b not found: ID does not exist" containerID="e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b"
Sep 14 10:43:14.341830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.341777    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b"} err="failed to get container status \"e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b\": rpc error: code = NotFound desc = could not find container \"e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b\": container with ID starting with e71d06d069bc3e3100ffbbda8f69d6f0360f5be8af31af513f8cabb2c6af7b8b not found: ID does not exist"
Sep 14 10:43:14.360246 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.360218    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"]
Sep 14 10:43:14.368008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.367979    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-f68032-openshift-default-6f79c44bc8-hdnmm"]
Sep 14 10:43:14.407250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.407223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:14.473840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.473803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:14.476227 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.476197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:14.499958 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.499929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:14.718228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.718189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:14.807935 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.807900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:14.909825 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.909793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:14.996006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:14.995982    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"]
Sep 14 10:43:14.998499 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:43:14.998460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8161d19_36ea_4edc_a29c_eaa7890d5238.slice/crio-f1aa94304992038215f80a72bd7a8b2427c49bcb85aa32a53f371a3719ac08a9 WatchSource:0}: Error finding container f1aa94304992038215f80a72bd7a8b2427c49bcb85aa32a53f371a3719ac08a9: Status 404 returned error can't find the container with id f1aa94304992038215f80a72bd7a8b2427c49bcb85aa32a53f371a3719ac08a9
Sep 14 10:43:15.339788 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.339701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx" event={"ID":"b8161d19-36ea-4edc-a29c-eaa7890d5238","Type":"ContainerStarted","Data":"f1aa94304992038215f80a72bd7a8b2427c49bcb85aa32a53f371a3719ac08a9"}
Sep 14 10:43:15.406157 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.406133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:15.506913 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.506877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:15.544318 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.544291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"]
Sep 14 10:43:15.547793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.547776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.550444 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.550415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5bbd7765\""
Sep 14 10:43:15.550563 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.550482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5bbd7765-dockercfg-r7rvt\""
Sep 14 10:43:15.560206 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.560179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"]
Sep 14 10:43:15.689158 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.689115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmbg\" (UniqueName: \"kubernetes.io/projected/e69500fd-3968-4e41-8231-c103c1aab76c-kube-api-access-lqmbg\") pod \"payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") " pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.689356 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.689242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69500fd-3968-4e41-8231-c103c1aab76c-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") " pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.790508 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.790470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmbg\" (UniqueName: \"kubernetes.io/projected/e69500fd-3968-4e41-8231-c103c1aab76c-kube-api-access-lqmbg\") pod \"payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") " pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.790713 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.790552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69500fd-3968-4e41-8231-c103c1aab76c-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") " pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.791251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.791224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69500fd-3968-4e41-8231-c103c1aab76c-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") " pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.798327 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.798303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmbg\" (UniqueName: \"kubernetes.io/projected/e69500fd-3968-4e41-8231-c103c1aab76c-kube-api-access-lqmbg\") pod \"payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") " pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.857854 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.857828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:15.921403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:15.921378    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c23ec3-adee-41bd-aa95-540e3b426d99" path="/var/lib/kubelet/pods/89c23ec3-adee-41bd-aa95-540e3b426d99/volumes"
Sep 14 10:43:16.004242 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.004167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:16.129405 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.129381    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"]
Sep 14 10:43:16.131246 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:43:16.131219    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69500fd_3968_4e41_8231_c103c1aab76c.slice/crio-cde282000fb2f0b58a84bf84695c3787fe8a4455bbdb9ec086aa64f8a6d15037 WatchSource:0}: Error finding container cde282000fb2f0b58a84bf84695c3787fe8a4455bbdb9ec086aa64f8a6d15037: Status 404 returned error can't find the container with id cde282000fb2f0b58a84bf84695c3787fe8a4455bbdb9ec086aa64f8a6d15037
Sep 14 10:43:16.347478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.347382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h" event={"ID":"e69500fd-3968-4e41-8231-c103c1aab76c","Type":"ContainerStarted","Data":"4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed"}
Sep 14 10:43:16.347478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.347444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:16.347478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.347467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h" event={"ID":"e69500fd-3968-4e41-8231-c103c1aab76c","Type":"ContainerStarted","Data":"cde282000fb2f0b58a84bf84695c3787fe8a4455bbdb9ec086aa64f8a6d15037"}
Sep 14 10:43:16.349173 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.349142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx" event={"ID":"b8161d19-36ea-4edc-a29c-eaa7890d5238","Type":"ContainerStarted","Data":"cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b"}
Sep 14 10:43:16.349287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.349236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:16.364208 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.364161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h" podStartSLOduration=1.364146482 podStartE2EDuration="1.364146482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:43:16.362053887 +0000 UTC m=+1084.917241812" watchObservedRunningTime="2026-09-14 10:43:16.364146482 +0000 UTC m=+1084.919334432"
Sep 14 10:43:16.380075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.380028    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx" podStartSLOduration=2.983720719 podStartE2EDuration="3.380013244s" totalImagesPullingTime="396.292525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:43:16.379502753 +0000 UTC m=+1084.934690680" watchObservedRunningTime="2026-09-14 10:43:16.380013244 +0000 UTC m=+1084.935201168"
Sep 14 10:43:16.724039 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:16.724007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:17.490721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:17.490685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:18.244746 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:18.244718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:18.987791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:18.987757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:22.360180 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:22.360152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:43:27.357068 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:27.357036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:43:28.069006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:28.068974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:43:28.742194 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:28.742156    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp" podUID="23bc6cb2-21dc-4968-9fc0-0825e7244a4e" containerName="main" containerID="cri-o://004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7" gracePeriod=45
Sep 14 10:43:29.004986 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.004929    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:43:29.114653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tmp-dir\") pod \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") "
Sep 14 10:43:29.114653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-home\") pod \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") "
Sep 14 10:43:29.114904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hgvq\" (UniqueName: \"kubernetes.io/projected/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-kube-api-access-9hgvq\") pod \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") "
Sep 14 10:43:29.114904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tls-certs\") pod \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") "
Sep 14 10:43:29.114904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-model-cache\") pod \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") "
Sep 14 10:43:29.114904 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-dshm\") pod \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\" (UID: \"23bc6cb2-21dc-4968-9fc0-0825e7244a4e\") "
Sep 14 10:43:29.115097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-home" (OuterVolumeSpecName: "home") pod "23bc6cb2-21dc-4968-9fc0-0825e7244a4e" (UID: "23bc6cb2-21dc-4968-9fc0-0825e7244a4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:29.115097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.114930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23bc6cb2-21dc-4968-9fc0-0825e7244a4e" (UID: "23bc6cb2-21dc-4968-9fc0-0825e7244a4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:29.115097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.115021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "23bc6cb2-21dc-4968-9fc0-0825e7244a4e" (UID: "23bc6cb2-21dc-4968-9fc0-0825e7244a4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:29.115234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.115193    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:29.115234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.115212    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:29.115234 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.115221    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:29.117015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.116989    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23bc6cb2-21dc-4968-9fc0-0825e7244a4e" (UID: "23bc6cb2-21dc-4968-9fc0-0825e7244a4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:43:29.117015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.117003    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-dshm" (OuterVolumeSpecName: "dshm") pod "23bc6cb2-21dc-4968-9fc0-0825e7244a4e" (UID: "23bc6cb2-21dc-4968-9fc0-0825e7244a4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:43:29.117149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.117084    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-kube-api-access-9hgvq" (OuterVolumeSpecName: "kube-api-access-9hgvq") pod "23bc6cb2-21dc-4968-9fc0-0825e7244a4e" (UID: "23bc6cb2-21dc-4968-9fc0-0825e7244a4e"). InnerVolumeSpecName "kube-api-access-9hgvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:43:29.215780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.215751    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:29.215780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.215776    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hgvq\" (UniqueName: \"kubernetes.io/projected/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-kube-api-access-9hgvq\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:29.215780 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.215786    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23bc6cb2-21dc-4968-9fc0-0825e7244a4e-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:43:29.417876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.417836    2563 generic.go:356] "Generic (PLEG): container finished" podID="23bc6cb2-21dc-4968-9fc0-0825e7244a4e" containerID="004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7" exitCode=0
Sep 14 10:43:29.418041 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.417886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"
Sep 14 10:43:29.418041 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.417922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp" event={"ID":"23bc6cb2-21dc-4968-9fc0-0825e7244a4e","Type":"ContainerDied","Data":"004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7"}
Sep 14 10:43:29.418041 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.417974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp" event={"ID":"23bc6cb2-21dc-4968-9fc0-0825e7244a4e","Type":"ContainerDied","Data":"1c5acae1ba812d97697845239519852ed5843c4a6b629f9c0aae16d8a935588f"}
Sep 14 10:43:29.418041 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.417996    2563 scope.go:117] "RemoveContainer" containerID="004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7"
Sep 14 10:43:29.429065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.429040    2563 scope.go:117] "RemoveContainer" containerID="004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7"
Sep 14 10:43:29.429458 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:43:29.429437    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7\": container with ID starting with 004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7 not found: ID does not exist" containerID="004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7"
Sep 14 10:43:29.429529 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.429467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7"} err="failed to get container status \"004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7\": rpc error: code = NotFound desc = could not find container \"004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7\": container with ID starting with 004c2027b18ccb5e1aa0ea1500f7c6343ee1f2a09ff29ca41ac345d8002612e7 not found: ID does not exist"
Sep 14 10:43:29.446754 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.446720    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"]
Sep 14 10:43:29.451272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.451235    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f68032/noten-model-f68032-kserve-54f988b699-r7fnp"]
Sep 14 10:43:29.918976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:29.918943    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23bc6cb2-21dc-4968-9fc0-0825e7244a4e" path="/var/lib/kubelet/pods/23bc6cb2-21dc-4968-9fc0-0825e7244a4e/volumes"
Sep 14 10:43:34.897137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:34.897105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:36.222514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:36.222485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:36.935386 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:36.935348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:37.688526 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:37.688486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:38.414184 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:38.414152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:39.079845 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:39.079813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:40.063169 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:40.063126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:40.576592 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:40.576533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:40.830786 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:40.830707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:40.975554 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:40.975519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:41.149588 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.149543    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"]
Sep 14 10:43:41.150052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.150033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23bc6cb2-21dc-4968-9fc0-0825e7244a4e" containerName="main"
Sep 14 10:43:41.150150 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.150054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23bc6cb2-21dc-4968-9fc0-0825e7244a4e" containerName="main"
Sep 14 10:43:41.150259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.150174    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23bc6cb2-21dc-4968-9fc0-0825e7244a4e" containerName="main"
Sep 14 10:43:41.153427 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.153405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.155896 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.155878    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 14 10:43:41.156005 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.155880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7bdcda-openshift-default-dockercfg-2rs5v\""
Sep 14 10:43:41.165842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.165812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"]
Sep 14 10:43:41.184828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.184792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:41.232530 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-socket\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232530 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-envoy\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-credential-socket\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-data\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nh2d\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-kube-api-access-9nh2d\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-certs\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istiod-ca-cert\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-podinfo\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.232850 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.232821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-token\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-socket\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-envoy\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-credential-socket\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-data\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nh2d\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-kube-api-access-9nh2d\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-certs\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istiod-ca-cert\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.333967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-podinfo\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.334124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.333970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-socket\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.334124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.334034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-data\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.334124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.334089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-credential-socket\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.334124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.334105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-token\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.334482 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.334257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-certs\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.334537 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.334513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istiod-ca-cert\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.336075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.336049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-envoy\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.336700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.336684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-podinfo\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.342195 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.342170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-token\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.342325 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.342312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nh2d\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-kube-api-access-9nh2d\") pod \"e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") " pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.464598 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.464491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:41.740627 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.740529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"]
Sep 14 10:43:41.743727 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:43:41.743701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d3eb7e_3abf_4bed_bad1_cf89c7dd77e0.slice/crio-bfb45d4a7802b50ce305345b86d861d75619a424a0dc83c75240355ad6764883 WatchSource:0}: Error finding container bfb45d4a7802b50ce305345b86d861d75619a424a0dc83c75240355ad6764883: Status 404 returned error can't find the container with id bfb45d4a7802b50ce305345b86d861d75619a424a0dc83c75240355ad6764883
Sep 14 10:43:41.746026 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.745998    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:43:41.746096 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.746064    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:43:41.746096 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.746091    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:43:41.881054 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:41.881014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:42.093060 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:42.092987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:42.455933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:42.455900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:42.487070 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:42.487040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" event={"ID":"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0","Type":"ContainerStarted","Data":"3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4"}
Sep 14 10:43:42.487211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:42.487091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" event={"ID":"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0","Type":"ContainerStarted","Data":"bfb45d4a7802b50ce305345b86d861d75619a424a0dc83c75240355ad6764883"}
Sep 14 10:43:42.510388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:42.510325    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podStartSLOduration=1.510305915 podStartE2EDuration="1.510305915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:43:42.507836656 +0000 UTC m=+1111.063024580" watchObservedRunningTime="2026-09-14 10:43:42.510305915 +0000 UTC m=+1111.065493842"
Sep 14 10:43:42.616076 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:42.616031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:43.297848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:43.297816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:43.465509 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:43.465472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:43.470637 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:43.470612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:43.493403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:43.493375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:43.494153 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:43.494132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:43:44.100638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:44.100600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:44.511866 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:44.511833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:44.620592 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:44.620537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:44.996362 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:44.996326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:45.752957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:45.752928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:46.617960 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:46.617932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:46.720165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:46.720133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:47.263025 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.262979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:47.360776 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.360740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:47.366718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.366690    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"]
Sep 14 10:43:47.370469 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.370454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.372704 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.372681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7bdcda-dockercfg-zc6hf\""
Sep 14 10:43:47.372830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.372754    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7bdcda\""
Sep 14 10:43:47.387204 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.387168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"]
Sep 14 10:43:47.393806 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.393779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f75w\" (UniqueName: \"kubernetes.io/projected/3b5f82b8-1414-418b-b895-c8c2a2125ba3-kube-api-access-6f75w\") pod \"payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.393933 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.393836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5f82b8-1414-418b-b895-c8c2a2125ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.494647 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.494610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f75w\" (UniqueName: \"kubernetes.io/projected/3b5f82b8-1414-418b-b895-c8c2a2125ba3-kube-api-access-6f75w\") pod \"payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.494813 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.494708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5f82b8-1414-418b-b895-c8c2a2125ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.495628 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.495603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5f82b8-1414-418b-b895-c8c2a2125ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.503377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.503346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f75w\" (UniqueName: \"kubernetes.io/projected/3b5f82b8-1414-418b-b895-c8c2a2125ba3-kube-api-access-6f75w\") pod \"payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.682420 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.682381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:47.694913 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.694876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:47.959600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:47.959550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"]
Sep 14 10:43:47.962361 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:43:47.962329    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5f82b8_1414_418b_b895_c8c2a2125ba3.slice/crio-850f189e9f9c22e237e553d3622553e5a6a34e24c4e28e9bd4b7ed50511638de WatchSource:0}: Error finding container 850f189e9f9c22e237e553d3622553e5a6a34e24c4e28e9bd4b7ed50511638de: Status 404 returned error can't find the container with id 850f189e9f9c22e237e553d3622553e5a6a34e24c4e28e9bd4b7ed50511638de
Sep 14 10:43:48.382980 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:48.382938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:48.520068 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:48.520032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts" event={"ID":"3b5f82b8-1414-418b-b895-c8c2a2125ba3","Type":"ContainerStarted","Data":"a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93"}
Sep 14 10:43:48.520240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:48.520083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts" event={"ID":"3b5f82b8-1414-418b-b895-c8c2a2125ba3","Type":"ContainerStarted","Data":"850f189e9f9c22e237e553d3622553e5a6a34e24c4e28e9bd4b7ed50511638de"}
Sep 14 10:43:48.520240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:48.520134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:43:48.538002 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:48.537948    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts" podStartSLOduration=1.537931 podStartE2EDuration="1.537931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:43:48.535524184 +0000 UTC m=+1117.090712111" watchObservedRunningTime="2026-09-14 10:43:48.537931 +0000 UTC m=+1117.093118930"
Sep 14 10:43:49.296415 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:49.296379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:50.020749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:50.020716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:50.850270 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:50.850238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:51.042243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:51.042214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:51.158653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:51.158615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:43:59.527375 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:43:59.527345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:44:03.592755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:03.592713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:44:05.099454 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:05.099422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:09.152786 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:09.152756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:09.409048 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:09.408965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:10.029328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:10.029269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:10.958862 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:10.958834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:12.121682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:12.121640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:12.914808 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:12.914767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:13.809302 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:13.809276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:13.872899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:13.872856    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": read tcp 10.134.0.2:44780->10.134.0.67:15021: read: connection reset by peer" start-of-body=
Sep 14 10:44:13.873035 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:13.872942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": read tcp 10.134.0.2:44780->10.134.0.67:15021: read: connection reset by peer"
Sep 14 10:44:14.610876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:14.610843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:14.676747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:14.676716    2563 generic.go:356] "Generic (PLEG): container finished" podID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerID="3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4" exitCode=0
Sep 14 10:44:14.676922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:14.676793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" event={"ID":"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0","Type":"ContainerDied","Data":"3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4"}
Sep 14 10:44:14.677263 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:14.677244    2563 scope.go:117] "RemoveContainer" containerID="3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4"
Sep 14 10:44:14.678510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:14.678481    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:44:14.678582 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:14.678553    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:44:14.678622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:14.678603    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:44:15.360298 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:15.360253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:15.683965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:15.683922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" event={"ID":"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0","Type":"ContainerStarted","Data":"653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5"}
Sep 14 10:44:16.464851 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:16.464817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:44:16.465247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:16.465138    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 14 10:44:16.465247 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:16.465182    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 14 10:44:17.465083 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:17.465043    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 14 10:44:17.465466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:17.465102    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 14 10:44:18.465539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:18.465503    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 14 10:44:18.465955 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:18.465558    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 14 10:44:19.042196 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:19.042162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:19.465696 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:19.465663    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 14 10:44:19.466064 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:19.465714    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 14 10:44:20.465120 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:20.465083    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 14 10:44:20.465300 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:20.465140    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 14 10:44:21.469312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:21.469278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:44:21.469814 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:21.469560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:44:21.470269 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:21.470253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:44:28.776458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:28.776423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:31.941460 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:31.941426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:34.088100 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:34.087986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:34.166326 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:34.166295    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:44:38.471888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:44:38.471855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:45:00.165124 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.165088    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9"]
Sep 14 10:45:00.167566 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.167547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9"
Sep 14 10:45:00.171225 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.171206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6vmmw\""
Sep 14 10:45:00.178240 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.178214    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9"]
Sep 14 10:45:00.179085 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.179062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9"
Sep 14 10:45:00.190524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.190498    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"]
Sep 14 10:45:00.193391 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.193306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"
Sep 14 10:45:00.204919 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.204887    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"]
Sep 14 10:45:00.207984 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.207960    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"
Sep 14 10:45:00.214088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.214056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"]
Sep 14 10:45:00.223374 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.223305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"]
Sep 14 10:45:00.223789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.223772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"
Sep 14 10:45:00.234664 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.234600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"
Sep 14 10:45:00.470661 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.470638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9"]
Sep 14 10:45:00.472415 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:45:00.472392    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc858bb9d_5247_4e2b_a081_5331a31141d9.slice/crio-81b6440a020f25e9a267490ac1d9fcedd02432761f3dece02a139108fc58fc3e WatchSource:0}: Error finding container 81b6440a020f25e9a267490ac1d9fcedd02432761f3dece02a139108fc58fc3e: Status 404 returned error can't find the container with id 81b6440a020f25e9a267490ac1d9fcedd02432761f3dece02a139108fc58fc3e
Sep 14 10:45:00.517261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.517238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"]
Sep 14 10:45:00.518960 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:45:00.518931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545875eb_4da3_41d6_8417_4432d9ff2fd0.slice/crio-afd60d88bed4682821497318248b55394b8dc2cf0ab25b029fd365719eecb816 WatchSource:0}: Error finding container afd60d88bed4682821497318248b55394b8dc2cf0ab25b029fd365719eecb816: Status 404 returned error can't find the container with id afd60d88bed4682821497318248b55394b8dc2cf0ab25b029fd365719eecb816
Sep 14 10:45:00.535855 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.535835    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"]
Sep 14 10:45:00.538136 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:45:00.538110    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ee48e6_82c5_45c4_a3e8_df83f596ce15.slice/crio-cfa76e048fbb45e90c484f6979fde39e416ad563f282cdb47426e24c9c7d73c9 WatchSource:0}: Error finding container cfa76e048fbb45e90c484f6979fde39e416ad563f282cdb47426e24c9c7d73c9: Status 404 returned error can't find the container with id cfa76e048fbb45e90c484f6979fde39e416ad563f282cdb47426e24c9c7d73c9
Sep 14 10:45:00.911825 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.911793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd" event={"ID":"545875eb-4da3-41d6-8417-4432d9ff2fd0","Type":"ContainerStarted","Data":"3c894c5c42da0ff111ef173b40ebf747c97f2b8fe421a20e73fad7a1742abb1a"}
Sep 14 10:45:00.911973 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.911899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd" event={"ID":"545875eb-4da3-41d6-8417-4432d9ff2fd0","Type":"ContainerStarted","Data":"afd60d88bed4682821497318248b55394b8dc2cf0ab25b029fd365719eecb816"}
Sep 14 10:45:00.913929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.913901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk" event={"ID":"b6ee48e6-82c5-45c4-a3e8-df83f596ce15","Type":"ContainerStarted","Data":"87d14f013a244e83ef0d5e83b035776699de97ced9c274db143447712018b2c8"}
Sep 14 10:45:00.914065 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.913949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk" event={"ID":"b6ee48e6-82c5-45c4-a3e8-df83f596ce15","Type":"ContainerStarted","Data":"cfa76e048fbb45e90c484f6979fde39e416ad563f282cdb47426e24c9c7d73c9"}
Sep 14 10:45:00.915408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.915385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9" event={"ID":"c858bb9d-5247-4e2b-a081-5331a31141d9","Type":"ContainerStarted","Data":"2acc8c80af49c3374c460f17122a83b7815a740defaa46f0a7d48a9889536b67"}
Sep 14 10:45:00.915499 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.915429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9" event={"ID":"c858bb9d-5247-4e2b-a081-5331a31141d9","Type":"ContainerStarted","Data":"81b6440a020f25e9a267490ac1d9fcedd02432761f3dece02a139108fc58fc3e"}
Sep 14 10:45:00.930185 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.929968    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd" podStartSLOduration=0.929952629 podStartE2EDuration="929.952629ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:45:00.929202611 +0000 UTC m=+1189.484390538" watchObservedRunningTime="2026-09-14 10:45:00.929952629 +0000 UTC m=+1189.485140555"
Sep 14 10:45:00.947159 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.947119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9" podStartSLOduration=0.947107538 podStartE2EDuration="947.107538ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:45:00.945168338 +0000 UTC m=+1189.500356276" watchObservedRunningTime="2026-09-14 10:45:00.947107538 +0000 UTC m=+1189.502295463"
Sep 14 10:45:00.964987 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:00.964949    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk" podStartSLOduration=0.964936699 podStartE2EDuration="964.936699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:45:00.963068153 +0000 UTC m=+1189.518256078" watchObservedRunningTime="2026-09-14 10:45:00.964936699 +0000 UTC m=+1189.520124624"
Sep 14 10:45:01.921923 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:01.921882    2563 generic.go:356] "Generic (PLEG): container finished" podID="b6ee48e6-82c5-45c4-a3e8-df83f596ce15" containerID="87d14f013a244e83ef0d5e83b035776699de97ced9c274db143447712018b2c8" exitCode=0
Sep 14 10:45:01.922314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:01.921964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk" event={"ID":"b6ee48e6-82c5-45c4-a3e8-df83f596ce15","Type":"ContainerDied","Data":"87d14f013a244e83ef0d5e83b035776699de97ced9c274db143447712018b2c8"}
Sep 14 10:45:01.923462 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:01.923439    2563 generic.go:356] "Generic (PLEG): container finished" podID="c858bb9d-5247-4e2b-a081-5331a31141d9" containerID="2acc8c80af49c3374c460f17122a83b7815a740defaa46f0a7d48a9889536b67" exitCode=0
Sep 14 10:45:01.923616 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:01.923517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9" event={"ID":"c858bb9d-5247-4e2b-a081-5331a31141d9","Type":"ContainerDied","Data":"2acc8c80af49c3374c460f17122a83b7815a740defaa46f0a7d48a9889536b67"}
Sep 14 10:45:01.925032 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:01.925011    2563 generic.go:356] "Generic (PLEG): container finished" podID="545875eb-4da3-41d6-8417-4432d9ff2fd0" containerID="3c894c5c42da0ff111ef173b40ebf747c97f2b8fe421a20e73fad7a1742abb1a" exitCode=0
Sep 14 10:45:01.925129 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:01.925086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd" event={"ID":"545875eb-4da3-41d6-8417-4432d9ff2fd0","Type":"ContainerDied","Data":"3c894c5c42da0ff111ef173b40ebf747c97f2b8fe421a20e73fad7a1742abb1a"}
Sep 14 10:45:03.060250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.060225    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"
Sep 14 10:45:03.107714 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.107690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"
Sep 14 10:45:03.110530 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.110512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9"
Sep 14 10:45:03.936472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.936451    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9"
Sep 14 10:45:03.936610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.936472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823045-4b5p9" event={"ID":"c858bb9d-5247-4e2b-a081-5331a31141d9","Type":"ContainerDied","Data":"81b6440a020f25e9a267490ac1d9fcedd02432761f3dece02a139108fc58fc3e"}
Sep 14 10:45:03.936610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.936511    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81b6440a020f25e9a267490ac1d9fcedd02432761f3dece02a139108fc58fc3e"
Sep 14 10:45:03.937789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.937773    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"
Sep 14 10:45:03.937886 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.937794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd" event={"ID":"545875eb-4da3-41d6-8417-4432d9ff2fd0","Type":"ContainerDied","Data":"afd60d88bed4682821497318248b55394b8dc2cf0ab25b029fd365719eecb816"}
Sep 14 10:45:03.937886 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.937827    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd60d88bed4682821497318248b55394b8dc2cf0ab25b029fd365719eecb816"
Sep 14 10:45:03.939211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.939187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk" event={"ID":"b6ee48e6-82c5-45c4-a3e8-df83f596ce15","Type":"ContainerDied","Data":"cfa76e048fbb45e90c484f6979fde39e416ad563f282cdb47426e24c9c7d73c9"}
Sep 14 10:45:03.939335 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.939228    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa76e048fbb45e90c484f6979fde39e416ad563f282cdb47426e24c9c7d73c9"
Sep 14 10:45:03.939335 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:03.939228    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"
Sep 14 10:45:06.666061 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666014    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"]
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666448    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ee48e6-82c5-45c4-a3e8-df83f596ce15" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ee48e6-82c5-45c4-a3e8-df83f596ce15" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666488    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545875eb-4da3-41d6-8417-4432d9ff2fd0" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666496    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="545875eb-4da3-41d6-8417-4432d9ff2fd0" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666505    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c858bb9d-5247-4e2b-a081-5331a31141d9" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666510    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c858bb9d-5247-4e2b-a081-5331a31141d9" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666603    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c858bb9d-5247-4e2b-a081-5331a31141d9" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666615    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ee48e6-82c5-45c4-a3e8-df83f596ce15" containerName="cleanup"
Sep 14 10:45:06.666665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.666624    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="545875eb-4da3-41d6-8417-4432d9ff2fd0" containerName="cleanup"
Sep 14 10:45:06.668855 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.668835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.673306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.673279    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5bbd7765\"/\"openshift-service-ca.crt\""
Sep 14 10:45:06.673793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.673758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5bbd7765\"/\"kube-root-ca.crt\""
Sep 14 10:45:06.673793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.673790    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5bbd7765\"/\"default-dockercfg-h7qwq\""
Sep 14 10:45:06.673986 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.673866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5bbd7765\"/\"e2e-lifecycle-model-5bbd7765-kserve-self-signed-certs\""
Sep 14 10:45:06.685328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.685305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"]
Sep 14 10:45:06.712717 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.712685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:06.796303 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.796278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-tmp-dir\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.796437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.796312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75vcc\" (UniqueName: \"kubernetes.io/projected/092b976a-7a40-4a27-90e5-ff007a39eedb-kube-api-access-75vcc\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.796437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.796351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-home\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.796437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.796382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-model-cache\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.796437 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.796407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092b976a-7a40-4a27-90e5-ff007a39eedb-tls-certs\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.796598 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.796533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-dshm\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.897955 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.897912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-home\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898130 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.897973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-model-cache\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898130 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.898012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092b976a-7a40-4a27-90e5-ff007a39eedb-tls-certs\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.898162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-dshm\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898283 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.898267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-tmp-dir\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.898315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75vcc\" (UniqueName: \"kubernetes.io/projected/092b976a-7a40-4a27-90e5-ff007a39eedb-kube-api-access-75vcc\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.898337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-home\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898435 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.898375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-model-cache\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.898614 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.898593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-tmp-dir\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.900387 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.900362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-dshm\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.900539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.900522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092b976a-7a40-4a27-90e5-ff007a39eedb-tls-certs\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.907144 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.907118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75vcc\" (UniqueName: \"kubernetes.io/projected/092b976a-7a40-4a27-90e5-ff007a39eedb-kube-api-access-75vcc\") pod \"e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") " pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:06.979476 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:06.979399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:07.255846 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:07.255816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"]
Sep 14 10:45:07.258135 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:45:07.258108    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod092b976a_7a40_4a27_90e5_ff007a39eedb.slice/crio-28dc270c173ea08c04ccd39ac4df5d0d138a01c520d6e383e6ffaa0946d08001 WatchSource:0}: Error finding container 28dc270c173ea08c04ccd39ac4df5d0d138a01c520d6e383e6ffaa0946d08001: Status 404 returned error can't find the container with id 28dc270c173ea08c04ccd39ac4df5d0d138a01c520d6e383e6ffaa0946d08001
Sep 14 10:45:07.963708 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:07.963678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl" event={"ID":"092b976a-7a40-4a27-90e5-ff007a39eedb","Type":"ContainerStarted","Data":"7bdde2eb91cbcec0ce357b729aa5627e0213f5f11f4f1fce761957738a128c12"}
Sep 14 10:45:07.963708 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:07.963714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl" event={"ID":"092b976a-7a40-4a27-90e5-ff007a39eedb","Type":"ContainerStarted","Data":"28dc270c173ea08c04ccd39ac4df5d0d138a01c520d6e383e6ffaa0946d08001"}
Sep 14 10:45:07.985879 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:07.985824    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl" podStartSLOduration=1.985797638 podStartE2EDuration="1.985797638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:45:07.985616526 +0000 UTC m=+1196.540804454" watchObservedRunningTime="2026-09-14 10:45:07.985797638 +0000 UTC m=+1196.540985566"
Sep 14 10:45:12.637836 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:12.637780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:16.780942 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:16.780904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:45:16.980489 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:16.980450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:16.980696 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:16.980540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:16.993090 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:16.993055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:17.022748 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:17.022716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:45:25.307475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:25.307444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:25.768506 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:25.768470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:26.585809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:26.585775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:27.505662 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:27.505625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:28.272072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:28.272031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:29.106750 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:29.106720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:30.001267 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:30.001233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:30.318173 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:30.318087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"]
Sep 14 10:45:30.332557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:30.332526    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"]
Sep 14 10:45:30.482498 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:30.482461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:30.905982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:30.905949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:31.698584 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:31.698540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:32.719912 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:32.719879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:41.032928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.032898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:41.068536 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.068506    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"]
Sep 14 10:45:41.068798 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.068771    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h" podUID="adbdff61-8069-4450-909a-d747f5f6f9bb" containerName="payload-processing" containerID="cri-o://a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699" gracePeriod=30
Sep 14 10:45:41.075921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.075892    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"]
Sep 14 10:45:41.076182 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.076156    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp" podUID="da661d50-70df-43bb-866f-ac1a4907c1e0" containerName="payload-pre-processing" containerID="cri-o://47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee" gracePeriod=30
Sep 14 10:45:41.345258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.345237    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:45:41.348053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.348031    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:45:41.433337 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.433288    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da661d50-70df-43bb-866f-ac1a4907c1e0-plugins-config-volume\") pod \"da661d50-70df-43bb-866f-ac1a4907c1e0\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") "
Sep 14 10:45:41.433551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.433374    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7mp8\" (UniqueName: \"kubernetes.io/projected/adbdff61-8069-4450-909a-d747f5f6f9bb-kube-api-access-n7mp8\") pod \"adbdff61-8069-4450-909a-d747f5f6f9bb\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") "
Sep 14 10:45:41.433551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.433404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdft4\" (UniqueName: \"kubernetes.io/projected/da661d50-70df-43bb-866f-ac1a4907c1e0-kube-api-access-pdft4\") pod \"da661d50-70df-43bb-866f-ac1a4907c1e0\" (UID: \"da661d50-70df-43bb-866f-ac1a4907c1e0\") "
Sep 14 10:45:41.433551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.433469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbdff61-8069-4450-909a-d747f5f6f9bb-plugins-config-volume\") pod \"adbdff61-8069-4450-909a-d747f5f6f9bb\" (UID: \"adbdff61-8069-4450-909a-d747f5f6f9bb\") "
Sep 14 10:45:41.433819 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.433786    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da661d50-70df-43bb-866f-ac1a4907c1e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da661d50-70df-43bb-866f-ac1a4907c1e0" (UID: "da661d50-70df-43bb-866f-ac1a4907c1e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:45:41.433937 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.433862    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbdff61-8069-4450-909a-d747f5f6f9bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "adbdff61-8069-4450-909a-d747f5f6f9bb" (UID: "adbdff61-8069-4450-909a-d747f5f6f9bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:45:41.433982 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.433944    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da661d50-70df-43bb-866f-ac1a4907c1e0-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:41.435699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.435678    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da661d50-70df-43bb-866f-ac1a4907c1e0-kube-api-access-pdft4" (OuterVolumeSpecName: "kube-api-access-pdft4") pod "da661d50-70df-43bb-866f-ac1a4907c1e0" (UID: "da661d50-70df-43bb-866f-ac1a4907c1e0"). InnerVolumeSpecName "kube-api-access-pdft4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:45:41.435782 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.435702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbdff61-8069-4450-909a-d747f5f6f9bb-kube-api-access-n7mp8" (OuterVolumeSpecName: "kube-api-access-n7mp8") pod "adbdff61-8069-4450-909a-d747f5f6f9bb" (UID: "adbdff61-8069-4450-909a-d747f5f6f9bb"). InnerVolumeSpecName "kube-api-access-n7mp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:45:41.534510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.534474    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mp8\" (UniqueName: \"kubernetes.io/projected/adbdff61-8069-4450-909a-d747f5f6f9bb-kube-api-access-n7mp8\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:41.534510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.534504    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdft4\" (UniqueName: \"kubernetes.io/projected/da661d50-70df-43bb-866f-ac1a4907c1e0-kube-api-access-pdft4\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:41.534510 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:41.534515    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbdff61-8069-4450-909a-d747f5f6f9bb-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:42.145801 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.145766    2563 generic.go:356] "Generic (PLEG): container finished" podID="adbdff61-8069-4450-909a-d747f5f6f9bb" containerID="a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699" exitCode=0
Sep 14 10:45:42.146140 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.145813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"
Sep 14 10:45:42.146140 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.145853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h" event={"ID":"adbdff61-8069-4450-909a-d747f5f6f9bb","Type":"ContainerDied","Data":"a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699"}
Sep 14 10:45:42.146140 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.145913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h" event={"ID":"adbdff61-8069-4450-909a-d747f5f6f9bb","Type":"ContainerDied","Data":"a1dfb23cd985a79ac30a8697619b4341d25a0c1cb6b88bafe472aaf8e27d460b"}
Sep 14 10:45:42.146140 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.145937    2563 scope.go:117] "RemoveContainer" containerID="a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699"
Sep 14 10:45:42.147329 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.147304    2563 generic.go:356] "Generic (PLEG): container finished" podID="da661d50-70df-43bb-866f-ac1a4907c1e0" containerID="47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee" exitCode=0
Sep 14 10:45:42.147513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.147342    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"
Sep 14 10:45:42.147630 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.147337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp" event={"ID":"da661d50-70df-43bb-866f-ac1a4907c1e0","Type":"ContainerDied","Data":"47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee"}
Sep 14 10:45:42.147692 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.147628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp" event={"ID":"da661d50-70df-43bb-866f-ac1a4907c1e0","Type":"ContainerDied","Data":"275dd6bafacdac55d74a9954e643aea25d65660d6d7eb72caf8b41162ab757e8"}
Sep 14 10:45:42.156901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.156878    2563 scope.go:117] "RemoveContainer" containerID="a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699"
Sep 14 10:45:42.157316 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:45:42.157290    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699\": container with ID starting with a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699 not found: ID does not exist" containerID="a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699"
Sep 14 10:45:42.157375 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.157325    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699"} err="failed to get container status \"a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699\": rpc error: code = NotFound desc = could not find container \"a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699\": container with ID starting with a2e337b9953d0c3b3021a0de64f2b6c1073b1c283b3b861c9bb54f9067b87699 not found: ID does not exist"
Sep 14 10:45:42.157375 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.157341    2563 scope.go:117] "RemoveContainer" containerID="47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee"
Sep 14 10:45:42.167847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.167826    2563 scope.go:117] "RemoveContainer" containerID="47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee"
Sep 14 10:45:42.168155 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:45:42.168094    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee\": container with ID starting with 47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee not found: ID does not exist" containerID="47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee"
Sep 14 10:45:42.168155 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.168129    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee"} err="failed to get container status \"47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee\": rpc error: code = NotFound desc = could not find container \"47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee\": container with ID starting with 47144bbb0128f0efe5c42a55e20a30f3edda5a155ef14ae1591e8084b7ae6cee not found: ID does not exist"
Sep 14 10:45:42.168683 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.168663    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"]
Sep 14 10:45:42.173947 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.173923    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-85d3ec-7c96c86b66-zcbqp"]
Sep 14 10:45:42.184964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.184906    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"]
Sep 14 10:45:42.187538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.187514    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-85d3ec-68799dcc5f-n8j5h"]
Sep 14 10:45:42.380264 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.380233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:42.597648 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:42.597531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:43.443135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:43.443099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:43.918348 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:43.918314    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbdff61-8069-4450-909a-d747f5f6f9bb" path="/var/lib/kubelet/pods/adbdff61-8069-4450-909a-d747f5f6f9bb/volumes"
Sep 14 10:45:43.918707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:43.918692    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da661d50-70df-43bb-866f-ac1a4907c1e0" path="/var/lib/kubelet/pods/da661d50-70df-43bb-866f-ac1a4907c1e0/volumes"
Sep 14 10:45:44.414827 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:44.414783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:45.326248 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.326217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:45.337519 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.337486    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk" podUID="84a47ee3-35ae-4602-a460-3d8f69d188ec" containerName="main" containerID="cri-o://3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c" gracePeriod=45
Sep 14 10:45:45.350695 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.350661    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m" podUID="cde1a739-f735-41d6-ac3e-5c01abba71fc" containerName="main" containerID="cri-o://9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b" gracePeriod=45
Sep 14 10:45:45.384511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.384479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:45:45.609561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.609538    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:45:45.629123 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.629099    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:45:45.776899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.776853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2vzh\" (UniqueName: \"kubernetes.io/projected/84a47ee3-35ae-4602-a460-3d8f69d188ec-kube-api-access-z2vzh\") pod \"84a47ee3-35ae-4602-a460-3d8f69d188ec\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") "
Sep 14 10:45:45.777114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.776910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-tmp-dir\") pod \"cde1a739-f735-41d6-ac3e-5c01abba71fc\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") "
Sep 14 10:45:45.777114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.776952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde1a739-f735-41d6-ac3e-5c01abba71fc-tls-certs\") pod \"cde1a739-f735-41d6-ac3e-5c01abba71fc\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") "
Sep 14 10:45:45.777114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.776980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-home\") pod \"84a47ee3-35ae-4602-a460-3d8f69d188ec\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") "
Sep 14 10:45:45.777114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777015    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-dshm\") pod \"cde1a739-f735-41d6-ac3e-5c01abba71fc\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") "
Sep 14 10:45:45.777114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-dshm\") pod \"84a47ee3-35ae-4602-a460-3d8f69d188ec\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") "
Sep 14 10:45:45.777114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-model-cache\") pod \"cde1a739-f735-41d6-ac3e-5c01abba71fc\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") "
Sep 14 10:45:45.777434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777125    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a47ee3-35ae-4602-a460-3d8f69d188ec-tls-certs\") pod \"84a47ee3-35ae-4602-a460-3d8f69d188ec\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") "
Sep 14 10:45:45.777434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-model-cache\") pod \"84a47ee3-35ae-4602-a460-3d8f69d188ec\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") "
Sep 14 10:45:45.777434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-home\") pod \"cde1a739-f735-41d6-ac3e-5c01abba71fc\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") "
Sep 14 10:45:45.777434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-tmp-dir\") pod \"84a47ee3-35ae-4602-a460-3d8f69d188ec\" (UID: \"84a47ee3-35ae-4602-a460-3d8f69d188ec\") "
Sep 14 10:45:45.777434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cde1a739-f735-41d6-ac3e-5c01abba71fc" (UID: "cde1a739-f735-41d6-ac3e-5c01abba71fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.777434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777267    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdvxm\" (UniqueName: \"kubernetes.io/projected/cde1a739-f735-41d6-ac3e-5c01abba71fc-kube-api-access-rdvxm\") pod \"cde1a739-f735-41d6-ac3e-5c01abba71fc\" (UID: \"cde1a739-f735-41d6-ac3e-5c01abba71fc\") "
Sep 14 10:45:45.777434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-home" (OuterVolumeSpecName: "home") pod "84a47ee3-35ae-4602-a460-3d8f69d188ec" (UID: "84a47ee3-35ae-4602-a460-3d8f69d188ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.777793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "84a47ee3-35ae-4602-a460-3d8f69d188ec" (UID: "84a47ee3-35ae-4602-a460-3d8f69d188ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.777793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-home" (OuterVolumeSpecName: "home") pod "cde1a739-f735-41d6-ac3e-5c01abba71fc" (UID: "cde1a739-f735-41d6-ac3e-5c01abba71fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.777910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777792    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84a47ee3-35ae-4602-a460-3d8f69d188ec" (UID: "84a47ee3-35ae-4602-a460-3d8f69d188ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.777910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cde1a739-f735-41d6-ac3e-5c01abba71fc" (UID: "cde1a739-f735-41d6-ac3e-5c01abba71fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.777910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777900    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.778224 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777921    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.778224 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777934    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.778224 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777944    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.778224 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.777954    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.779539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.779504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a47ee3-35ae-4602-a460-3d8f69d188ec-kube-api-access-z2vzh" (OuterVolumeSpecName: "kube-api-access-z2vzh") pod "84a47ee3-35ae-4602-a460-3d8f69d188ec" (UID: "84a47ee3-35ae-4602-a460-3d8f69d188ec"). InnerVolumeSpecName "kube-api-access-z2vzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:45:45.779682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.779606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde1a739-f735-41d6-ac3e-5c01abba71fc-kube-api-access-rdvxm" (OuterVolumeSpecName: "kube-api-access-rdvxm") pod "cde1a739-f735-41d6-ac3e-5c01abba71fc" (UID: "cde1a739-f735-41d6-ac3e-5c01abba71fc"). InnerVolumeSpecName "kube-api-access-rdvxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:45:45.779773 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.779756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde1a739-f735-41d6-ac3e-5c01abba71fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cde1a739-f735-41d6-ac3e-5c01abba71fc" (UID: "cde1a739-f735-41d6-ac3e-5c01abba71fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:45:45.779827 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.779795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-dshm" (OuterVolumeSpecName: "dshm") pod "cde1a739-f735-41d6-ac3e-5c01abba71fc" (UID: "cde1a739-f735-41d6-ac3e-5c01abba71fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.780038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.780013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-dshm" (OuterVolumeSpecName: "dshm") pod "84a47ee3-35ae-4602-a460-3d8f69d188ec" (UID: "84a47ee3-35ae-4602-a460-3d8f69d188ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:45:45.780110 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.780031    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a47ee3-35ae-4602-a460-3d8f69d188ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84a47ee3-35ae-4602-a460-3d8f69d188ec" (UID: "84a47ee3-35ae-4602-a460-3d8f69d188ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:45:45.879095 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.879055    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2vzh\" (UniqueName: \"kubernetes.io/projected/84a47ee3-35ae-4602-a460-3d8f69d188ec-kube-api-access-z2vzh\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.879095 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.879091    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde1a739-f735-41d6-ac3e-5c01abba71fc-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.879312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.879103    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.879312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.879115    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a47ee3-35ae-4602-a460-3d8f69d188ec-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.879312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.879124    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde1a739-f735-41d6-ac3e-5c01abba71fc-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.879312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.879134    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a47ee3-35ae-4602-a460-3d8f69d188ec-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:45.879312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:45.879144    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvxm\" (UniqueName: \"kubernetes.io/projected/cde1a739-f735-41d6-ac3e-5c01abba71fc-kube-api-access-rdvxm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:45:46.105459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.105420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:46.170735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.170643    2563 generic.go:356] "Generic (PLEG): container finished" podID="84a47ee3-35ae-4602-a460-3d8f69d188ec" containerID="3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c" exitCode=0
Sep 14 10:45:46.170735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.170691    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"
Sep 14 10:45:46.170976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.170728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk" event={"ID":"84a47ee3-35ae-4602-a460-3d8f69d188ec","Type":"ContainerDied","Data":"3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c"}
Sep 14 10:45:46.170976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.170777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk" event={"ID":"84a47ee3-35ae-4602-a460-3d8f69d188ec","Type":"ContainerDied","Data":"8294bdfbe1a11f7cfb9aac828c01178d830a927ad7c71ab2925c28da4c6313de"}
Sep 14 10:45:46.170976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.170800    2563 scope.go:117] "RemoveContainer" containerID="3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c"
Sep 14 10:45:46.172239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.172214    2563 generic.go:356] "Generic (PLEG): container finished" podID="cde1a739-f735-41d6-ac3e-5c01abba71fc" containerID="9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b" exitCode=0
Sep 14 10:45:46.172361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.172264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"
Sep 14 10:45:46.172361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.172286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m" event={"ID":"cde1a739-f735-41d6-ac3e-5c01abba71fc","Type":"ContainerDied","Data":"9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b"}
Sep 14 10:45:46.172361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.172327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m" event={"ID":"cde1a739-f735-41d6-ac3e-5c01abba71fc","Type":"ContainerDied","Data":"8d8087491770d3e1455db17c431760e3666ba84c020198f68249b2d93e725228"}
Sep 14 10:45:46.179741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.179725    2563 scope.go:117] "RemoveContainer" containerID="3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c"
Sep 14 10:45:46.180006 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:45:46.179989    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c\": container with ID starting with 3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c not found: ID does not exist" containerID="3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c"
Sep 14 10:45:46.180052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.180012    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c"} err="failed to get container status \"3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c\": rpc error: code = NotFound desc = could not find container \"3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c\": container with ID starting with 3c2363f06d4c8c07bd7154c4d85920da8e1d9f0c4ab0882c5d1297968796cc2c not found: ID does not exist"
Sep 14 10:45:46.180052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.180027    2563 scope.go:117] "RemoveContainer" containerID="9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b"
Sep 14 10:45:46.188072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.188057    2563 scope.go:117] "RemoveContainer" containerID="9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b"
Sep 14 10:45:46.188332 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:45:46.188315    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b\": container with ID starting with 9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b not found: ID does not exist" containerID="9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b"
Sep 14 10:45:46.188408 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.188338    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b"} err="failed to get container status \"9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b\": rpc error: code = NotFound desc = could not find container \"9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b\": container with ID starting with 9a5ec6117e7d3aa609629879c32c6e73d6a8a287cd28f2dfd6e5f549c0bbf22b not found: ID does not exist"
Sep 14 10:45:46.189172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.189154    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"]
Sep 14 10:45:46.193653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.193634    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/premium-w0-85d3ec-kserve-7c7f688dbb-5kcbk"]
Sep 14 10:45:46.203474 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.203442    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"]
Sep 14 10:45:46.208149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.208128    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-85d3ec/simulator-w0-85d3ec-kserve-58f8f4dc9f-9hr4m"]
Sep 14 10:45:46.825475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:46.825446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:47.747470 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:47.747442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:47.918361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:47.918328    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a47ee3-35ae-4602-a460-3d8f69d188ec" path="/var/lib/kubelet/pods/84a47ee3-35ae-4602-a460-3d8f69d188ec/volumes"
Sep 14 10:45:47.918729 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:47.918715    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde1a739-f735-41d6-ac3e-5c01abba71fc" path="/var/lib/kubelet/pods/cde1a739-f735-41d6-ac3e-5c01abba71fc/volumes"
Sep 14 10:45:48.327490 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:48.327455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:48.397173 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:48.397143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:48.491959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:48.491930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:48.611429 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:48.611348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:52.625431 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:52.625373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:53.724799 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:53.724768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:53.845832 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:53.845796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:45:54.645753 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:45:54.645723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:14.072911 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:14.072874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:46:21.510670 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:21.510628    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"]
Sep 14 10:46:21.541733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:21.541694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:26.403991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:26.403957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:26.507678 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:26.507643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:27.589852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:27.589816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:31.499095 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:31.499067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:32.281879 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:32.281843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:33.179884 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:33.179846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:34.024090 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:34.024060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:34.914675 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:34.914642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:35.835042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:35.835013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:36.529475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.529436    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l" podUID="f08e2680-7df1-4401-b559-7c1fdbf37a11" containerName="main" containerID="cri-o://a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776" gracePeriod=45
Sep 14 10:46:36.644601 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.644559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:36.769521 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.769497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:46:36.782319 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.782264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:36.798404 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.798370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:46:36.924863 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.924826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-home\") pod \"f08e2680-7df1-4401-b559-7c1fdbf37a11\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") "
Sep 14 10:46:36.925038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.924881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2srwm\" (UniqueName: \"kubernetes.io/projected/f08e2680-7df1-4401-b559-7c1fdbf37a11-kube-api-access-2srwm\") pod \"f08e2680-7df1-4401-b559-7c1fdbf37a11\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") "
Sep 14 10:46:36.925038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.924918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e2680-7df1-4401-b559-7c1fdbf37a11-tls-certs\") pod \"f08e2680-7df1-4401-b559-7c1fdbf37a11\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") "
Sep 14 10:46:36.925038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-tmp-dir\") pod \"f08e2680-7df1-4401-b559-7c1fdbf37a11\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") "
Sep 14 10:46:36.925216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-home" (OuterVolumeSpecName: "home") pod "f08e2680-7df1-4401-b559-7c1fdbf37a11" (UID: "f08e2680-7df1-4401-b559-7c1fdbf37a11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:46:36.925216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-model-cache\") pod \"f08e2680-7df1-4401-b559-7c1fdbf37a11\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") "
Sep 14 10:46:36.925216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-dshm\") pod \"f08e2680-7df1-4401-b559-7c1fdbf37a11\" (UID: \"f08e2680-7df1-4401-b559-7c1fdbf37a11\") "
Sep 14 10:46:36.925385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f08e2680-7df1-4401-b559-7c1fdbf37a11" (UID: "f08e2680-7df1-4401-b559-7c1fdbf37a11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:46:36.925469 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925447    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-model-cache" (OuterVolumeSpecName: "model-cache") pod "f08e2680-7df1-4401-b559-7c1fdbf37a11" (UID: "f08e2680-7df1-4401-b559-7c1fdbf37a11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:46:36.925724 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925705    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:36.925839 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925762    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:36.925839 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.925775    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:36.927345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.927323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08e2680-7df1-4401-b559-7c1fdbf37a11-kube-api-access-2srwm" (OuterVolumeSpecName: "kube-api-access-2srwm") pod "f08e2680-7df1-4401-b559-7c1fdbf37a11" (UID: "f08e2680-7df1-4401-b559-7c1fdbf37a11"). InnerVolumeSpecName "kube-api-access-2srwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:46:36.927774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.927757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-dshm" (OuterVolumeSpecName: "dshm") pod "f08e2680-7df1-4401-b559-7c1fdbf37a11" (UID: "f08e2680-7df1-4401-b559-7c1fdbf37a11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:46:36.927862 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:36.927774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08e2680-7df1-4401-b559-7c1fdbf37a11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f08e2680-7df1-4401-b559-7c1fdbf37a11" (UID: "f08e2680-7df1-4401-b559-7c1fdbf37a11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:46:37.026911 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.026874    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e2680-7df1-4401-b559-7c1fdbf37a11-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:37.026911 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.026910    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2srwm\" (UniqueName: \"kubernetes.io/projected/f08e2680-7df1-4401-b559-7c1fdbf37a11-kube-api-access-2srwm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:37.026911 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.026925    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e2680-7df1-4401-b559-7c1fdbf37a11-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:37.442598 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.442546    2563 generic.go:356] "Generic (PLEG): container finished" podID="f08e2680-7df1-4401-b559-7c1fdbf37a11" containerID="a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776" exitCode=0
Sep 14 10:46:37.442747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.442642    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"
Sep 14 10:46:37.442747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.442645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l" event={"ID":"f08e2680-7df1-4401-b559-7c1fdbf37a11","Type":"ContainerDied","Data":"a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776"}
Sep 14 10:46:37.442747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.442704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l" event={"ID":"f08e2680-7df1-4401-b559-7c1fdbf37a11","Type":"ContainerDied","Data":"58e471cf33d4cc13c724abf7686e2eceda46617cdb556d820036f9df86219dd9"}
Sep 14 10:46:37.442747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.442722    2563 scope.go:117] "RemoveContainer" containerID="a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776"
Sep 14 10:46:37.451923 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.451906    2563 scope.go:117] "RemoveContainer" containerID="a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776"
Sep 14 10:46:37.452181 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:46:37.452162    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776\": container with ID starting with a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776 not found: ID does not exist" containerID="a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776"
Sep 14 10:46:37.452232 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.452190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776"} err="failed to get container status \"a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776\": rpc error: code = NotFound desc = could not find container \"a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776\": container with ID starting with a87ffa6cba256bd4791a3663b0642e81d6c233e651432862613e6803153c7776 not found: ID does not exist"
Sep 14 10:46:37.466035 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.466012    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"]
Sep 14 10:46:37.477641 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.477621    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/e2e-conflict-a-588c9108-kserve-5cfbbc8df5-6fp4l"]
Sep 14 10:46:37.917967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:37.917939    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08e2680-7df1-4401-b559-7c1fdbf37a11" path="/var/lib/kubelet/pods/f08e2680-7df1-4401-b559-7c1fdbf37a11/volumes"
Sep 14 10:46:46.119701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.119669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:46.159375 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.159341    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"]
Sep 14 10:46:46.159687 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.159659    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h" podUID="e69500fd-3968-4e41-8231-c103c1aab76c" containerName="payload-processing" containerID="cri-o://4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed" gracePeriod=30
Sep 14 10:46:46.222622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.222563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:46.322072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.322039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:46.401853 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.401809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:46.413410 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.413389    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:46:46.489911 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.489879    2563 generic.go:356] "Generic (PLEG): container finished" podID="e69500fd-3968-4e41-8231-c103c1aab76c" containerID="4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed" exitCode=0
Sep 14 10:46:46.490102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.489929    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"
Sep 14 10:46:46.490102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.489966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h" event={"ID":"e69500fd-3968-4e41-8231-c103c1aab76c","Type":"ContainerDied","Data":"4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed"}
Sep 14 10:46:46.490102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.490019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h" event={"ID":"e69500fd-3968-4e41-8231-c103c1aab76c","Type":"ContainerDied","Data":"cde282000fb2f0b58a84bf84695c3787fe8a4455bbdb9ec086aa64f8a6d15037"}
Sep 14 10:46:46.490102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.490047    2563 scope.go:117] "RemoveContainer" containerID="4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed"
Sep 14 10:46:46.497551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.497524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqmbg\" (UniqueName: \"kubernetes.io/projected/e69500fd-3968-4e41-8231-c103c1aab76c-kube-api-access-lqmbg\") pod \"e69500fd-3968-4e41-8231-c103c1aab76c\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") "
Sep 14 10:46:46.497740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.497723    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69500fd-3968-4e41-8231-c103c1aab76c-plugins-config-volume\") pod \"e69500fd-3968-4e41-8231-c103c1aab76c\" (UID: \"e69500fd-3968-4e41-8231-c103c1aab76c\") "
Sep 14 10:46:46.498088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.498064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69500fd-3968-4e41-8231-c103c1aab76c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e69500fd-3968-4e41-8231-c103c1aab76c" (UID: "e69500fd-3968-4e41-8231-c103c1aab76c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:46:46.499351 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.499333    2563 scope.go:117] "RemoveContainer" containerID="4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed"
Sep 14 10:46:46.499665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.499644    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69500fd-3968-4e41-8231-c103c1aab76c-kube-api-access-lqmbg" (OuterVolumeSpecName: "kube-api-access-lqmbg") pod "e69500fd-3968-4e41-8231-c103c1aab76c" (UID: "e69500fd-3968-4e41-8231-c103c1aab76c"). InnerVolumeSpecName "kube-api-access-lqmbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:46:46.499733 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:46:46.499644    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed\": container with ID starting with 4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed not found: ID does not exist" containerID="4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed"
Sep 14 10:46:46.499733 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.499695    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed"} err="failed to get container status \"4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed\": rpc error: code = NotFound desc = could not find container \"4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed\": container with ID starting with 4195b4cb2996ac05864712c88fcdd824801e86ebfb3f59101fae3be94f0f81ed not found: ID does not exist"
Sep 14 10:46:46.598452 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.598417    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmbg\" (UniqueName: \"kubernetes.io/projected/e69500fd-3968-4e41-8231-c103c1aab76c-kube-api-access-lqmbg\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:46.598452 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.598448    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69500fd-3968-4e41-8231-c103c1aab76c-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:46:46.811375 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.811344    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"]
Sep 14 10:46:46.815190 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.815165    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bbd7765-5569b7ff9c-wpf7h"]
Sep 14 10:46:46.822187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:46.822168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:47.658129 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:47.658093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:47.919305 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:47.919231    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69500fd-3968-4e41-8231-c103c1aab76c" path="/var/lib/kubelet/pods/e69500fd-3968-4e41-8231-c103c1aab76c/volumes"
Sep 14 10:46:48.607414 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:48.607382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:49.466821 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:49.466791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:49.567443 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:49.567407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:50.206897 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:50.206863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:51.030538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:51.030508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:52.683838 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:52.683802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:54.463390 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:54.463356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"]
Sep 14 10:46:54.557646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:54.557596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:56.442011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.441977    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"]
Sep 14 10:46:56.442372 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442359    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbdff61-8069-4450-909a-d747f5f6f9bb" containerName="payload-processing"
Sep 14 10:46:56.442417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442373    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbdff61-8069-4450-909a-d747f5f6f9bb" containerName="payload-processing"
Sep 14 10:46:56.442417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442383    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da661d50-70df-43bb-866f-ac1a4907c1e0" containerName="payload-pre-processing"
Sep 14 10:46:56.442417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442389    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da661d50-70df-43bb-866f-ac1a4907c1e0" containerName="payload-pre-processing"
Sep 14 10:46:56.442417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69500fd-3968-4e41-8231-c103c1aab76c" containerName="payload-processing"
Sep 14 10:46:56.442417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442417    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69500fd-3968-4e41-8231-c103c1aab76c" containerName="payload-processing"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442424    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08e2680-7df1-4401-b559-7c1fdbf37a11" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442429    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08e2680-7df1-4401-b559-7c1fdbf37a11" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442436    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a47ee3-35ae-4602-a460-3d8f69d188ec" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442441    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a47ee3-35ae-4602-a460-3d8f69d188ec" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442452    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde1a739-f735-41d6-ac3e-5c01abba71fc" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442456    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde1a739-f735-41d6-ac3e-5c01abba71fc" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442517    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde1a739-f735-41d6-ac3e-5c01abba71fc" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442524    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbdff61-8069-4450-909a-d747f5f6f9bb" containerName="payload-processing"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442531    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69500fd-3968-4e41-8231-c103c1aab76c" containerName="payload-processing"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442537    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da661d50-70df-43bb-866f-ac1a4907c1e0" containerName="payload-pre-processing"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442542    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08e2680-7df1-4401-b559-7c1fdbf37a11" containerName="main"
Sep 14 10:46:56.442596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.442550    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a47ee3-35ae-4602-a460-3d8f69d188ec" containerName="main"
Sep 14 10:46:56.446970 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.446953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.452178 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.452157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"auth-test-model-7bdcda-kserve-self-signed-certs\""
Sep 14 10:46:56.452478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.452453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"openshift-service-ca.crt\""
Sep 14 10:46:56.452719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.452705    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"kube-root-ca.crt\""
Sep 14 10:46:56.453621 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.453601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"default-dockercfg-b6p85\""
Sep 14 10:46:56.486833 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.486806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-tmp-dir\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.486954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.486854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093759ae-b088-4802-b36f-0d2c014b42e5-tls-certs\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.486954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.486929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-home\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.487057 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.486983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbsx\" (UniqueName: \"kubernetes.io/projected/093759ae-b088-4802-b36f-0d2c014b42e5-kube-api-access-jjbsx\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.487103 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.487076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-model-cache\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.487153 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.487139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-dshm\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.493397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.493373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"]
Sep 14 10:46:56.514402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.514356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:46:56.588695 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.588656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-tmp-dir\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.588695 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.588696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093759ae-b088-4802-b36f-0d2c014b42e5-tls-certs\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.588927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.588751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-home\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.588927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.588801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbsx\" (UniqueName: \"kubernetes.io/projected/093759ae-b088-4802-b36f-0d2c014b42e5-kube-api-access-jjbsx\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.588927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.588840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-model-cache\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.588927 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.588891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-dshm\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.589168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.589141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-tmp-dir\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.589239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.589175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-home\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.589292 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.589263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-model-cache\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.591174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.591155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-dshm\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.591550 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.591530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093759ae-b088-4802-b36f-0d2c014b42e5-tls-certs\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.600014 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.599993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbsx\" (UniqueName: \"kubernetes.io/projected/093759ae-b088-4802-b36f-0d2c014b42e5-kube-api-access-jjbsx\") pod \"auth-test-model-7bdcda-kserve-5f975998d7-gq67c\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:56.755779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:56.755721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:46:57.026381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:57.026353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"]
Sep 14 10:46:57.028238 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:46:57.028208    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093759ae_b088_4802_b36f_0d2c014b42e5.slice/crio-6fca66c02d0c6a4f2a8c07bf5015cf1932a189782f35a3d1eb443bb23250f2f7 WatchSource:0}: Error finding container 6fca66c02d0c6a4f2a8c07bf5015cf1932a189782f35a3d1eb443bb23250f2f7: Status 404 returned error can't find the container with id 6fca66c02d0c6a4f2a8c07bf5015cf1932a189782f35a3d1eb443bb23250f2f7
Sep 14 10:46:57.548796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:57.548760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c" event={"ID":"093759ae-b088-4802-b36f-0d2c014b42e5","Type":"ContainerStarted","Data":"207b394d2a53c10957909bdfae7006076edf8ac96f44f86111abf049058f2421"}
Sep 14 10:46:57.548796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:57.548808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c" event={"ID":"093759ae-b088-4802-b36f-0d2c014b42e5","Type":"ContainerStarted","Data":"6fca66c02d0c6a4f2a8c07bf5015cf1932a189782f35a3d1eb443bb23250f2f7"}
Sep 14 10:46:57.575900 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:46:57.575848    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c" podStartSLOduration=1.575832406 podStartE2EDuration="1.575832406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:46:57.574110618 +0000 UTC m=+1306.129298543" watchObservedRunningTime="2026-09-14 10:46:57.575832406 +0000 UTC m=+1306.131020324"
Sep 14 10:47:02.493497 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:02.493409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:47:06.756230 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:06.756189    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:47:06.756230 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:06.756242    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:47:06.768548 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:06.768523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:47:06.883791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:06.883752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:07.611930 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:07.611902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:47:09.482962 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.482923    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl" podUID="092b976a-7a40-4a27-90e5-ff007a39eedb" containerName="main" containerID="cri-o://7bdde2eb91cbcec0ce357b729aa5627e0213f5f11f4f1fce761957738a128c12" gracePeriod=45
Sep 14 10:47:09.618174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.618136    2563 generic.go:356] "Generic (PLEG): container finished" podID="092b976a-7a40-4a27-90e5-ff007a39eedb" containerID="7bdde2eb91cbcec0ce357b729aa5627e0213f5f11f4f1fce761957738a128c12" exitCode=0
Sep 14 10:47:09.618305 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.618207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl" event={"ID":"092b976a-7a40-4a27-90e5-ff007a39eedb","Type":"ContainerDied","Data":"7bdde2eb91cbcec0ce357b729aa5627e0213f5f11f4f1fce761957738a128c12"}
Sep 14 10:47:09.743294 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.743230    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:47:09.812122 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-home\") pod \"092b976a-7a40-4a27-90e5-ff007a39eedb\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") "
Sep 14 10:47:09.812122 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-dshm\") pod \"092b976a-7a40-4a27-90e5-ff007a39eedb\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") "
Sep 14 10:47:09.812370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-model-cache\") pod \"092b976a-7a40-4a27-90e5-ff007a39eedb\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") "
Sep 14 10:47:09.812370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812234    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75vcc\" (UniqueName: \"kubernetes.io/projected/092b976a-7a40-4a27-90e5-ff007a39eedb-kube-api-access-75vcc\") pod \"092b976a-7a40-4a27-90e5-ff007a39eedb\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") "
Sep 14 10:47:09.812370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-tmp-dir\") pod \"092b976a-7a40-4a27-90e5-ff007a39eedb\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") "
Sep 14 10:47:09.812370 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092b976a-7a40-4a27-90e5-ff007a39eedb-tls-certs\") pod \"092b976a-7a40-4a27-90e5-ff007a39eedb\" (UID: \"092b976a-7a40-4a27-90e5-ff007a39eedb\") "
Sep 14 10:47:09.812562 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812384    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-home" (OuterVolumeSpecName: "home") pod "092b976a-7a40-4a27-90e5-ff007a39eedb" (UID: "092b976a-7a40-4a27-90e5-ff007a39eedb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:47:09.812562 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-model-cache" (OuterVolumeSpecName: "model-cache") pod "092b976a-7a40-4a27-90e5-ff007a39eedb" (UID: "092b976a-7a40-4a27-90e5-ff007a39eedb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:47:09.812672 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "092b976a-7a40-4a27-90e5-ff007a39eedb" (UID: "092b976a-7a40-4a27-90e5-ff007a39eedb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:47:09.812761 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812743    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:47:09.812815 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812764    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:47:09.812815 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.812777    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:47:09.814602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.814553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-dshm" (OuterVolumeSpecName: "dshm") pod "092b976a-7a40-4a27-90e5-ff007a39eedb" (UID: "092b976a-7a40-4a27-90e5-ff007a39eedb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:47:09.814602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.814589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092b976a-7a40-4a27-90e5-ff007a39eedb-kube-api-access-75vcc" (OuterVolumeSpecName: "kube-api-access-75vcc") pod "092b976a-7a40-4a27-90e5-ff007a39eedb" (UID: "092b976a-7a40-4a27-90e5-ff007a39eedb"). InnerVolumeSpecName "kube-api-access-75vcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:47:09.814749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.814663    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092b976a-7a40-4a27-90e5-ff007a39eedb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "092b976a-7a40-4a27-90e5-ff007a39eedb" (UID: "092b976a-7a40-4a27-90e5-ff007a39eedb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:47:09.913954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.913916    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75vcc\" (UniqueName: \"kubernetes.io/projected/092b976a-7a40-4a27-90e5-ff007a39eedb-kube-api-access-75vcc\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:47:09.913954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.913944    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092b976a-7a40-4a27-90e5-ff007a39eedb-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:47:09.913954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:09.913954    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092b976a-7a40-4a27-90e5-ff007a39eedb-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:47:10.624470 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:10.624432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl" event={"ID":"092b976a-7a40-4a27-90e5-ff007a39eedb","Type":"ContainerDied","Data":"28dc270c173ea08c04ccd39ac4df5d0d138a01c520d6e383e6ffaa0946d08001"}
Sep 14 10:47:10.624860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:10.624447    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"
Sep 14 10:47:10.624860 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:10.624476    2563 scope.go:117] "RemoveContainer" containerID="7bdde2eb91cbcec0ce357b729aa5627e0213f5f11f4f1fce761957738a128c12"
Sep 14 10:47:10.642421 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:10.642398    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"]
Sep 14 10:47:10.646962 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:10.646941    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5bbd7765/e2e-lifecycle-model-5bbd7765-kserve-5fbbf6d4b7-rc9tl"]
Sep 14 10:47:11.919179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:11.919147    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092b976a-7a40-4a27-90e5-ff007a39eedb" path="/var/lib/kubelet/pods/092b976a-7a40-4a27-90e5-ff007a39eedb/volumes"
Sep 14 10:47:29.413292 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:29.413264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:29.515438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:29.515408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:29.908639 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:29.908607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:30.176912 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.176838    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"]
Sep 14 10:47:30.177264 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.177252    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092b976a-7a40-4a27-90e5-ff007a39eedb" containerName="main"
Sep 14 10:47:30.177302 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.177266    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="092b976a-7a40-4a27-90e5-ff007a39eedb" containerName="main"
Sep 14 10:47:30.177340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.177326    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="092b976a-7a40-4a27-90e5-ff007a39eedb" containerName="main"
Sep 14 10:47:30.181773 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.181754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.183815 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.183798    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1910e9\"/\"distinct-2-w4-1910e9-kserve-self-signed-certs\""
Sep 14 10:47:30.196374 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.196342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"]
Sep 14 10:47:30.204865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.204845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:30.289496 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.289472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec046b7-d46b-49e6-8239-90e44ed50284-tls-certs\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.289620 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.289503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-tmp-dir\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.289620 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.289544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-home\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.289695 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.289618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-model-cache\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.289741 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.289721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfpv\" (UniqueName: \"kubernetes.io/projected/dec046b7-d46b-49e6-8239-90e44ed50284-kube-api-access-7jfpv\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.289776 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.289759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-dshm\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.390783 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.390752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec046b7-d46b-49e6-8239-90e44ed50284-tls-certs\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.390783 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.390789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-tmp-dir\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.391044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.390849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-home\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.391044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.390871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-model-cache\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.391044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.390969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfpv\" (UniqueName: \"kubernetes.io/projected/dec046b7-d46b-49e6-8239-90e44ed50284-kube-api-access-7jfpv\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.391044 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.390994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-dshm\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.391264 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.391242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-tmp-dir\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.391324 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.391304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-home\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.391366 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.391324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-model-cache\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.393207 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.393188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-dshm\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.393387 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.393371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec046b7-d46b-49e6-8239-90e44ed50284-tls-certs\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.402171 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.402149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfpv\" (UniqueName: \"kubernetes.io/projected/dec046b7-d46b-49e6-8239-90e44ed50284-kube-api-access-7jfpv\") pod \"distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") " pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.490867 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.490809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:30.766034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:30.766011    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"]
Sep 14 10:47:30.768556 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:47:30.768527    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddec046b7_d46b_49e6_8239_90e44ed50284.slice/crio-4c8aaaceeae5d06307b0dbfd0d1eb1ff90fb7c0cf11592ed07f709315784af09 WatchSource:0}: Error finding container 4c8aaaceeae5d06307b0dbfd0d1eb1ff90fb7c0cf11592ed07f709315784af09: Status 404 returned error can't find the container with id 4c8aaaceeae5d06307b0dbfd0d1eb1ff90fb7c0cf11592ed07f709315784af09
Sep 14 10:47:31.732947 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:31.732903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn" event={"ID":"dec046b7-d46b-49e6-8239-90e44ed50284","Type":"ContainerStarted","Data":"adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b"}
Sep 14 10:47:31.733379 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:31.732955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn" event={"ID":"dec046b7-d46b-49e6-8239-90e44ed50284","Type":"ContainerStarted","Data":"4c8aaaceeae5d06307b0dbfd0d1eb1ff90fb7c0cf11592ed07f709315784af09"}
Sep 14 10:47:31.734098 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:31.734077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:31.755397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:31.755341    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn" podStartSLOduration=1.755321174 podStartE2EDuration="1.755321174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:47:31.751047721 +0000 UTC m=+1340.306235646" watchObservedRunningTime="2026-09-14 10:47:31.755321174 +0000 UTC m=+1340.310509100"
Sep 14 10:47:32.193920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:32.193884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:32.414172 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:32.414141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:47:37.886828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:37.886754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:37.987788 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:37.987758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:38.149287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:38.149199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:39.488162 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:39.488133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:39.580431 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:39.580405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:40.240997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:40.240969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:40.323586 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:40.323552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:40.491638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:40.491541    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:40.492008 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:40.491638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:40.503545 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:40.503528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:40.788537 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:40.788464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:47:41.618847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:41.618810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:47.924952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:47.924915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:48.045338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:48.045304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:48.670872 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:48.670837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:53.280782 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:53.280748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:47:54.999361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:54.999332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:55.144886 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:55.144854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:47:55.255942 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:47:55.255868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:04.896555 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:04.896523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:05.683239 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:05.683212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:05.908059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:05.908020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:16.027006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:16.026970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:48:26.862423 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:26.862387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:33.511536 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:33.511500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:33.609488 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:33.609457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:33.841024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:33.840926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:34.119309 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.119236    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"]
Sep 14 10:48:34.122898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.122881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.126161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.126141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d220da-openshift-default-dockercfg-ltj7s\""
Sep 14 10:48:34.150364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.150343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"]
Sep 14 10:48:34.175765 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.175738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:34.185821 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.185793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-token\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.185937 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.185895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-credential-socket\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.185986 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.185938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-certs\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.185986 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.185982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istiod-ca-cert\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.186053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.186000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9br7\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-kube-api-access-x9br7\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.186053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.186026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-envoy\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.186053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.186044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-data\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.186161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.186065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-podinfo\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.186161 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.186117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-socket\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287039 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-credential-socket\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287039 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-certs\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istiod-ca-cert\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9br7\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-kube-api-access-x9br7\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-envoy\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-data\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-podinfo\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-socket\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-token\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287694 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-credential-socket\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287694 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-certs\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287694 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-data\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.287857 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.287697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-socket\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.288127 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.288105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istiod-ca-cert\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.289476 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.289459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-envoy\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.289887 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.289866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-podinfo\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.302867 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.302845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-token\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.302965 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.302912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9br7\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-kube-api-access-x9br7\") pod \"e2e-api-a-d220da-openshift-default-74b6789978-29pww\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") " pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.432453 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.432420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:34.441705 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.441674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:34.724734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.724656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"]
Sep 14 10:48:34.728018 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:48:34.727992    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ba1c2a_5736_423f_9cc7_b6ed4bddc20e.slice/crio-6daa4ef1f1df387cb6f9428c12ea9e10c637afc23cf16a102cf6edb2fe314713 WatchSource:0}: Error finding container 6daa4ef1f1df387cb6f9428c12ea9e10c637afc23cf16a102cf6edb2fe314713: Status 404 returned error can't find the container with id 6daa4ef1f1df387cb6f9428c12ea9e10c637afc23cf16a102cf6edb2fe314713
Sep 14 10:48:34.730228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.730198    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:48:34.730294 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.730257    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:48:34.730294 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:34.730285    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:48:35.062602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:35.062504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" event={"ID":"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e","Type":"ContainerStarted","Data":"4a682265323896270a2387adcc776c43f9899897ff17650c8fa299a93f67ae55"}
Sep 14 10:48:35.062602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:35.062561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" event={"ID":"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e","Type":"ContainerStarted","Data":"6daa4ef1f1df387cb6f9428c12ea9e10c637afc23cf16a102cf6edb2fe314713"}
Sep 14 10:48:35.088389 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:35.088342    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podStartSLOduration=1.088325899 podStartE2EDuration="1.088325899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:35.086922382 +0000 UTC m=+1403.642110308" watchObservedRunningTime="2026-09-14 10:48:35.088325899 +0000 UTC m=+1403.643513829"
Sep 14 10:48:35.433538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:35.433506    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:35.438426 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:35.438405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:35.600039 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:35.600003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:36.067869 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:36.067835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:36.068737 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:36.068716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:48:36.704549 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:36.704514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:37.015609 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:37.015504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:37.103713 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:37.103677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:37.708844 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:37.708806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:38.489097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:38.489065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:39.390550 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:39.390516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:40.128895 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:40.128869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:40.871872 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:40.871843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:41.774035 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.774006    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"]
Sep 14 10:48:41.777436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.777409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.780028 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.779992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d220da-serving-cert\""
Sep 14 10:48:41.790354 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.790330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"]
Sep 14 10:48:41.816255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.816228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:41.863998 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.863960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca2ed28-148f-43ca-8a79-84d5d17d8076-openshift-service-ca\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.864138 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.864117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-maas-api-tls\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.864189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.864149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-metrics-certs\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.864189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.864185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvsbm\" (UniqueName: \"kubernetes.io/projected/5ca2ed28-148f-43ca-8a79-84d5d17d8076-kube-api-access-gvsbm\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.917486 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.917466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:41.965217 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.965181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-maas-api-tls\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.965217 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.965221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-metrics-certs\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.965466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.965252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvsbm\" (UniqueName: \"kubernetes.io/projected/5ca2ed28-148f-43ca-8a79-84d5d17d8076-kube-api-access-gvsbm\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.965466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.965281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca2ed28-148f-43ca-8a79-84d5d17d8076-openshift-service-ca\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.965989 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.965968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca2ed28-148f-43ca-8a79-84d5d17d8076-openshift-service-ca\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.967845 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.967823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-metrics-certs\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.967954 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.967935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-maas-api-tls\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:41.973423 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:41.973402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvsbm\" (UniqueName: \"kubernetes.io/projected/5ca2ed28-148f-43ca-8a79-84d5d17d8076-kube-api-access-gvsbm\") pod \"maas-api-e2e-api-a-d220da-6474c79c45-w9k98\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:42.016906 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:42.016881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:42.087946 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:42.087884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:42.331023 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:42.330963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:42.360500 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:42.360470    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"]
Sep 14 10:48:42.363595 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:48:42.363548    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca2ed28_148f_43ca_8a79_84d5d17d8076.slice/crio-4e5b0ed680eff87a12e2b023f23d5b9fbdae5a89bedca72c7e5d30fc6f569375 WatchSource:0}: Error finding container 4e5b0ed680eff87a12e2b023f23d5b9fbdae5a89bedca72c7e5d30fc6f569375: Status 404 returned error can't find the container with id 4e5b0ed680eff87a12e2b023f23d5b9fbdae5a89bedca72c7e5d30fc6f569375
Sep 14 10:48:42.365002 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:42.364982    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 10:48:42.434024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:42.433994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:43.105799 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:43.105762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98" event={"ID":"5ca2ed28-148f-43ca-8a79-84d5d17d8076","Type":"ContainerStarted","Data":"712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a"}
Sep 14 10:48:43.106183 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:43.105817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98" event={"ID":"5ca2ed28-148f-43ca-8a79-84d5d17d8076","Type":"ContainerStarted","Data":"4e5b0ed680eff87a12e2b023f23d5b9fbdae5a89bedca72c7e5d30fc6f569375"}
Sep 14 10:48:43.106183 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:43.105872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:48:43.124800 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:43.124745    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98" podStartSLOduration=1.5674022669999998 podStartE2EDuration="2.124725918s" totalImagesPullingTime="557.323651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:43.124623286 +0000 UTC m=+1411.679811212" watchObservedRunningTime="2026-09-14 10:48:43.124725918 +0000 UTC m=+1411.679913843"
Sep 14 10:48:43.477325 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:43.477293    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:48:46.579988 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:46.579956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:48:49.116291 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:48:49.116262    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:49:02.834552 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:02.834518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:04.655080 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:04.655044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:05.382626 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:05.382594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:06.319492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:06.319457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:07.034747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:07.034711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:07.109716 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:07.109679    2563 patch_prober.go:28] interesting pod/e2e-api-a-d220da-openshift-default-74b6789978-29pww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:49:07.109889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:07.109753    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:49:07.135340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:07.135285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:07.496602 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:07.496544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:07.816125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:07.816044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:07.899605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:07.899552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:08.255646 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.255621    2563 generic.go:356] "Generic (PLEG): container finished" podID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerID="4a682265323896270a2387adcc776c43f9899897ff17650c8fa299a93f67ae55" exitCode=0
Sep 14 10:49:08.255798 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.255688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" event={"ID":"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e","Type":"ContainerDied","Data":"4a682265323896270a2387adcc776c43f9899897ff17650c8fa299a93f67ae55"}
Sep 14 10:49:08.256115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.256098    2563 scope.go:117] "RemoveContainer" containerID="4a682265323896270a2387adcc776c43f9899897ff17650c8fa299a93f67ae55"
Sep 14 10:49:08.257323 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.257296    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:49:08.257402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.257377    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:49:08.257440 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.257409    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:49:08.270197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.270175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:08.680388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:08.680357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:09.263178 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:09.263143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" event={"ID":"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e","Type":"ContainerStarted","Data":"bd71a15a3e71a3d64139d36daafe27de3393401798a076b09d6d742d156aec0a"}
Sep 14 10:49:09.432967 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:09.432939    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:49:09.433146 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:09.433091    2563 patch_prober.go:28] interesting pod/e2e-api-a-d220da-openshift-default-74b6789978-29pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 14 10:49:09.433146 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:09.433130    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 14 10:49:09.810373 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:09.810333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:10.434276 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:10.433820    2563 patch_prober.go:28] interesting pod/e2e-api-a-d220da-openshift-default-74b6789978-29pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 14 10:49:10.434276 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:10.433884    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 14 10:49:10.732710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:10.732608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:11.433197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:11.433157    2563 patch_prober.go:28] interesting pod/e2e-api-a-d220da-openshift-default-74b6789978-29pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 14 10:49:11.433712 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:11.433229    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 14 10:49:11.470980 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:11.470944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:11.604291 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:11.604254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:11.710655 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:11.710539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:11.796349 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:11.796317    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:49:12.433464 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:12.433427    2563 patch_prober.go:28] interesting pod/e2e-api-a-d220da-openshift-default-74b6789978-29pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 14 10:49:12.434058 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:12.434023    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 14 10:49:12.552707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:12.552676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:13.433055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:13.433002    2563 patch_prober.go:28] interesting pod/e2e-api-a-d220da-openshift-default-74b6789978-29pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 14 10:49:13.433258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:13.433085    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 14 10:49:13.527038 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:13.526982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:14.433654 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:14.433619    2563 patch_prober.go:28] interesting pod/e2e-api-a-d220da-openshift-default-74b6789978-29pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 14 10:49:14.433961 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:14.433933    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 14 10:49:14.647252 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:14.647222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:15.437368 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:15.437329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:49:15.437682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:15.437655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:49:15.438268 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:15.438249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:49:15.589744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:15.589709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:16.393409 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:16.393375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:17.225314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.225270    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"]
Sep 14 10:49:17.227936 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.227917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.236686 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.236667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ea2835-serving-cert\""
Sep 14 10:49:17.255983 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.255959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"]
Sep 14 10:49:17.270865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.270840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:17.312839 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.312804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2rs\" (UniqueName: \"kubernetes.io/projected/89311ae2-10b5-497b-9ad8-037283ba1893-kube-api-access-kj2rs\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.313002 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.312854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-metrics-certs\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.313002 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.312942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-maas-api-tls\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.313002 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.312968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89311ae2-10b5-497b-9ad8-037283ba1893-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.372806 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.372777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:17.414090 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.414057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2rs\" (UniqueName: \"kubernetes.io/projected/89311ae2-10b5-497b-9ad8-037283ba1893-kube-api-access-kj2rs\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.414090 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.414106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-metrics-certs\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.414626 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.414172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-maas-api-tls\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.414626 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.414193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89311ae2-10b5-497b-9ad8-037283ba1893-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.415105 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.415012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89311ae2-10b5-497b-9ad8-037283ba1893-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.416773 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.416752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-metrics-certs\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.416886 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.416785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-maas-api-tls\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.428622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.428596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2rs\" (UniqueName: \"kubernetes.io/projected/89311ae2-10b5-497b-9ad8-037283ba1893-kube-api-access-kj2rs\") pod \"maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.538064 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.537995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:17.816391 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.816363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"]
Sep 14 10:49:17.818529 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:49:17.818495    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89311ae2_10b5_497b_9ad8_037283ba1893.slice/crio-5b99a85f7420ba62650ec07d0dba320641cb7560b78df00485052447d9b81544 WatchSource:0}: Error finding container 5b99a85f7420ba62650ec07d0dba320641cb7560b78df00485052447d9b81544: Status 404 returned error can't find the container with id 5b99a85f7420ba62650ec07d0dba320641cb7560b78df00485052447d9b81544
Sep 14 10:49:17.929892 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:17.929856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:18.030722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:18.030692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:18.309878 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:18.309851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph" event={"ID":"89311ae2-10b5-497b-9ad8-037283ba1893","Type":"ContainerStarted","Data":"a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97"}
Sep 14 10:49:18.310012 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:18.309896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph" event={"ID":"89311ae2-10b5-497b-9ad8-037283ba1893","Type":"ContainerStarted","Data":"5b99a85f7420ba62650ec07d0dba320641cb7560b78df00485052447d9b81544"}
Sep 14 10:49:18.310012 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:18.309987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:24.321521 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:24.321493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:49:24.340976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:24.340928    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph" podStartSLOduration=6.956269616 podStartE2EDuration="7.340909536s" totalImagesPullingTime="384.63992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:18.328547364 +0000 UTC m=+1446.883735292" watchObservedRunningTime="2026-09-14 10:49:24.340909536 +0000 UTC m=+1452.896097461"
Sep 14 10:49:41.176479 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:41.176441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:41.277557 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:41.277515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:41.381325 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:41.381290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:49:41.411362 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:41.411335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:42.157268 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:42.157228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:43.001995 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:43.001963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:43.931789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:43.931759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:45.017851 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:45.017820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:45.837054 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:45.837023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:46.868165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:46.868132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:47.701176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:47.701113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:48.487350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:48.487314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:49.377638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:49.377563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:50.096624 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:50.096564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:50.815144 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:50.815113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:49:59.865972 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:59.865934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"]
Sep 14 10:49:59.866365 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:59.866184    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98" podUID="5ca2ed28-148f-43ca-8a79-84d5d17d8076" containerName="maas-api" containerID="cri-o://712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a" gracePeriod=30
Sep 14 10:49:59.914928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:49:59.914875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:00.112215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.112185    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:50:00.210402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.210377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-metrics-certs\") pod \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") "
Sep 14 10:50:00.210590 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.210472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvsbm\" (UniqueName: \"kubernetes.io/projected/5ca2ed28-148f-43ca-8a79-84d5d17d8076-kube-api-access-gvsbm\") pod \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") "
Sep 14 10:50:00.210590 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.210515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca2ed28-148f-43ca-8a79-84d5d17d8076-openshift-service-ca\") pod \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") "
Sep 14 10:50:00.210590 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.210557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-maas-api-tls\") pod \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\" (UID: \"5ca2ed28-148f-43ca-8a79-84d5d17d8076\") "
Sep 14 10:50:00.210899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.210869    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca2ed28-148f-43ca-8a79-84d5d17d8076-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ca2ed28-148f-43ca-8a79-84d5d17d8076" (UID: "5ca2ed28-148f-43ca-8a79-84d5d17d8076"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:50:00.212539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.212508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ca2ed28-148f-43ca-8a79-84d5d17d8076" (UID: "5ca2ed28-148f-43ca-8a79-84d5d17d8076"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:00.212653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.212596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ca2ed28-148f-43ca-8a79-84d5d17d8076" (UID: "5ca2ed28-148f-43ca-8a79-84d5d17d8076"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:00.212653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.212603    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca2ed28-148f-43ca-8a79-84d5d17d8076-kube-api-access-gvsbm" (OuterVolumeSpecName: "kube-api-access-gvsbm") pod "5ca2ed28-148f-43ca-8a79-84d5d17d8076" (UID: "5ca2ed28-148f-43ca-8a79-84d5d17d8076"). InnerVolumeSpecName "kube-api-access-gvsbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:00.312115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.312075    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvsbm\" (UniqueName: \"kubernetes.io/projected/5ca2ed28-148f-43ca-8a79-84d5d17d8076-kube-api-access-gvsbm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:00.312115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.312109    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca2ed28-148f-43ca-8a79-84d5d17d8076-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:00.312115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.312119    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:00.312115 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.312127    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ca2ed28-148f-43ca-8a79-84d5d17d8076-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:00.534372 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.534286    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ca2ed28-148f-43ca-8a79-84d5d17d8076" containerID="712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a" exitCode=0
Sep 14 10:50:00.534372 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.534328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"
Sep 14 10:50:00.534372 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.534358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98" event={"ID":"5ca2ed28-148f-43ca-8a79-84d5d17d8076","Type":"ContainerDied","Data":"712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a"}
Sep 14 10:50:00.534681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.534409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98" event={"ID":"5ca2ed28-148f-43ca-8a79-84d5d17d8076","Type":"ContainerDied","Data":"4e5b0ed680eff87a12e2b023f23d5b9fbdae5a89bedca72c7e5d30fc6f569375"}
Sep 14 10:50:00.534681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.534437    2563 scope.go:117] "RemoveContainer" containerID="712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a"
Sep 14 10:50:00.543757 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.543738    2563 scope.go:117] "RemoveContainer" containerID="712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a"
Sep 14 10:50:00.543999 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:50:00.543982    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a\": container with ID starting with 712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a not found: ID does not exist" containerID="712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a"
Sep 14 10:50:00.544048 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.544005    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a"} err="failed to get container status \"712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a\": rpc error: code = NotFound desc = could not find container \"712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a\": container with ID starting with 712081d67133882b5aeb58c2891b37c3663d21074f71b78502c864ade87a908a not found: ID does not exist"
Sep 14 10:50:00.556691 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.556672    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"]
Sep 14 10:50:00.560701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.560681    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d220da-6474c79c45-w9k98"]
Sep 14 10:50:00.568698 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.568680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:00.644486 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:00.644463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:01.402413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:01.402372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:01.919765 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:01.919736    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ca2ed28-148f-43ca-8a79-84d5d17d8076" path="/var/lib/kubelet/pods/5ca2ed28-148f-43ca-8a79-84d5d17d8076/volumes"
Sep 14 10:50:02.242215 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:02.242189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:03.135340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:03.135309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:03.968594 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:03.968541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:04.538179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:04.538149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:04.971658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:04.971618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:05.700063 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:05.700011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:06.423443 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:06.423409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:07.125095 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:07.125062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:07.232491 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:07.232461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:07.337710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:07.337672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:07.356143 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:07.356111    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"]
Sep 14 10:50:07.356345 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:07.356324    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy" containerID="cri-o://bd71a15a3e71a3d64139d36daafe27de3393401798a076b09d6d742d156aec0a" gracePeriod=30
Sep 14 10:50:07.432654 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:07.432500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:08.329851 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:08.329819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:50:08.493848 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:08.493809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:09.443707 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:09.443655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:09.547925 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:09.547885    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"]
Sep 14 10:50:09.561355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:09.561323    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"]
Sep 14 10:50:09.575481 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:09.575455    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"]
Sep 14 10:50:09.651093 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:09.651058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:10.302703 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:10.302653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:11.204730 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:11.204702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:11.917941 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:11.917896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:12.599359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.599330    2563 generic.go:356] "Generic (PLEG): container finished" podID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerID="bd71a15a3e71a3d64139d36daafe27de3393401798a076b09d6d742d156aec0a" exitCode=0
Sep 14 10:50:12.599689 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.599403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" event={"ID":"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e","Type":"ContainerDied","Data":"bd71a15a3e71a3d64139d36daafe27de3393401798a076b09d6d742d156aec0a"}
Sep 14 10:50:12.599689 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.599485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww" event={"ID":"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e","Type":"ContainerDied","Data":"6daa4ef1f1df387cb6f9428c12ea9e10c637afc23cf16a102cf6edb2fe314713"}
Sep 14 10:50:12.599689 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.599503    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6daa4ef1f1df387cb6f9428c12ea9e10c637afc23cf16a102cf6edb2fe314713"
Sep 14 10:50:12.599689 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.599517    2563 scope.go:117] "RemoveContainer" containerID="4a682265323896270a2387adcc776c43f9899897ff17650c8fa299a93f67ae55"
Sep 14 10:50:12.600402 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.600316    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:50:12.626651 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-token\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-envoy\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626734 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-socket\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istiod-ca-cert\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9br7\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-kube-api-access-x9br7\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-data\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-podinfo\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626862    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-credential-socket\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.626952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626905    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-certs\") pod \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\" (UID: \"38ba1c2a-5736-423f-9cc7-b6ed4bddc20e\") "
Sep 14 10:50:12.627094 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.626977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:12.627162 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627139    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-data" (OuterVolumeSpecName: "istio-data") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:12.627162 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:12.627343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:50:12.627343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627278    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.627343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:12.627343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627297    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.627343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627311    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.627343 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.627324    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.628833 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.628814    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-token" (OuterVolumeSpecName: "istio-token") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:12.629265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.629243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-kube-api-access-x9br7" (OuterVolumeSpecName: "kube-api-access-x9br7") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "kube-api-access-x9br7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:12.629458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.629434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:12.629458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.629437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" (UID: "38ba1c2a-5736-423f-9cc7-b6ed4bddc20e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:50:12.728729 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.728700    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9br7\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-kube-api-access-x9br7\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.728729 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.728724    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.728729 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.728733    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.728928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.728741    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:12.728928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:12.728748    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:13.605350 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:13.605317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"
Sep 14 10:50:13.648288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:13.648261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:13.665011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:13.664983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"]
Sep 14 10:50:13.669539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:13.669517    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d220da-openshift-default-74b6789978-29pww"]
Sep 14 10:50:13.831128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:13.831098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:13.918674 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:13.918638    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" path="/var/lib/kubelet/pods/38ba1c2a-5736-423f-9cc7-b6ed4bddc20e/volumes"
Sep 14 10:50:15.143619 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:15.143587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:15.307360 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:15.307329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:16.454261 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:16.454228    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"]
Sep 14 10:50:16.535384 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:16.535353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:17.889380 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:17.889342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:21.057613 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.057559    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"]
Sep 14 10:50:21.061517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.061484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:21.063211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.063185    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1910e9-29823045-lpgtd"]
Sep 14 10:50:21.093467 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.093434    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"]
Sep 14 10:50:21.093764 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.093738    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh" podUID="8b9515cd-3937-48b8-b230-52a9d74fbb63" containerName="payload-processing" containerID="cri-o://6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b" gracePeriod=30
Sep 14 10:50:21.156273 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.156232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:21.256256 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.256225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:21.338282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.338253    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:50:21.360833 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.360774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:21.402063 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.402022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9515cd-3937-48b8-b230-52a9d74fbb63-plugins-config-volume\") pod \"8b9515cd-3937-48b8-b230-52a9d74fbb63\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") "
Sep 14 10:50:21.402238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.402135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltnv9\" (UniqueName: \"kubernetes.io/projected/8b9515cd-3937-48b8-b230-52a9d74fbb63-kube-api-access-ltnv9\") pod \"8b9515cd-3937-48b8-b230-52a9d74fbb63\" (UID: \"8b9515cd-3937-48b8-b230-52a9d74fbb63\") "
Sep 14 10:50:21.402489 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.402462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9515cd-3937-48b8-b230-52a9d74fbb63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b9515cd-3937-48b8-b230-52a9d74fbb63" (UID: "8b9515cd-3937-48b8-b230-52a9d74fbb63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:50:21.404299 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.404278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9515cd-3937-48b8-b230-52a9d74fbb63-kube-api-access-ltnv9" (OuterVolumeSpecName: "kube-api-access-ltnv9") pod "8b9515cd-3937-48b8-b230-52a9d74fbb63" (UID: "8b9515cd-3937-48b8-b230-52a9d74fbb63"). InnerVolumeSpecName "kube-api-access-ltnv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:21.503186 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.503143    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltnv9\" (UniqueName: \"kubernetes.io/projected/8b9515cd-3937-48b8-b230-52a9d74fbb63-kube-api-access-ltnv9\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:21.503186 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.503182    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9515cd-3937-48b8-b230-52a9d74fbb63-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:21.639511 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.639482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:21.647704 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.647678    2563 generic.go:356] "Generic (PLEG): container finished" podID="8b9515cd-3937-48b8-b230-52a9d74fbb63" containerID="6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b" exitCode=0
Sep 14 10:50:21.647830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.647726    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"
Sep 14 10:50:21.647830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.647760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh" event={"ID":"8b9515cd-3937-48b8-b230-52a9d74fbb63","Type":"ContainerDied","Data":"6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b"}
Sep 14 10:50:21.647830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.647815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh" event={"ID":"8b9515cd-3937-48b8-b230-52a9d74fbb63","Type":"ContainerDied","Data":"7e948504b44929791e96caa4fff4a581d9ce4af1c2b9b7cde71aaa39b0c0681b"}
Sep 14 10:50:21.647980 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.647836    2563 scope.go:117] "RemoveContainer" containerID="6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b"
Sep 14 10:50:21.656796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.656780    2563 scope.go:117] "RemoveContainer" containerID="6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b"
Sep 14 10:50:21.657047 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:50:21.657028    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b\": container with ID starting with 6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b not found: ID does not exist" containerID="6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b"
Sep 14 10:50:21.657103 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.657055    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b"} err="failed to get container status \"6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b\": rpc error: code = NotFound desc = could not find container \"6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b\": container with ID starting with 6af6aae498c48a60688e1e6cd0eabe71c6607a2d1c41f17fd1d344b10f7dd29b not found: ID does not exist"
Sep 14 10:50:21.670710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.670680    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"]
Sep 14 10:50:21.673216 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.673198    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1910e9-5fdc8d8bd5-zvjfh"]
Sep 14 10:50:21.774270 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.774247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:21.919403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.919326    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545875eb-4da3-41d6-8417-4432d9ff2fd0" path="/var/lib/kubelet/pods/545875eb-4da3-41d6-8417-4432d9ff2fd0/volumes"
Sep 14 10:50:21.919712 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:21.919699    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9515cd-3937-48b8-b230-52a9d74fbb63" path="/var/lib/kubelet/pods/8b9515cd-3937-48b8-b230-52a9d74fbb63/volumes"
Sep 14 10:50:22.513683 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:22.513647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:23.417673 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:23.417640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:24.230286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.230253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:24.567668 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.567561    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn" podUID="dec046b7-d46b-49e6-8239-90e44ed50284" containerName="main" containerID="cri-o://adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b" gracePeriod=45
Sep 14 10:50:24.580932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.580901    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p" podUID="edaecc21-86b5-4e62-bcdd-90076b9ee8fd" containerName="main" containerID="cri-o://24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0" gracePeriod=45
Sep 14 10:50:24.594910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.594866    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl" podUID="8c2a5b6e-0294-4b34-9f8c-bc6219e04467" containerName="main" containerID="cri-o://83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e" gracePeriod=45
Sep 14 10:50:24.848590 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.848543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:50:24.876016 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.875992    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:50:24.879395 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.879373    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:50:24.932882 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.932848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-home\") pod \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") "
Sep 14 10:50:24.933069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.932900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tls-certs\") pod \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") "
Sep 14 10:50:24.933069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.932930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-home\") pod \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") "
Sep 14 10:50:24.933069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.932961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-home\") pod \"dec046b7-d46b-49e6-8239-90e44ed50284\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") "
Sep 14 10:50:24.933069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.932993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96cb6\" (UniqueName: \"kubernetes.io/projected/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-kube-api-access-96cb6\") pod \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") "
Sep 14 10:50:24.933069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-model-cache\") pod \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") "
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tls-certs\") pod \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") "
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933103    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-home" (OuterVolumeSpecName: "home") pod "edaecc21-86b5-4e62-bcdd-90076b9ee8fd" (UID: "edaecc21-86b5-4e62-bcdd-90076b9ee8fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933119    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-model-cache\") pod \"dec046b7-d46b-49e6-8239-90e44ed50284\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") "
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933150    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-kube-api-access-gfth8\") pod \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") "
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933173    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-dshm\") pod \"dec046b7-d46b-49e6-8239-90e44ed50284\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") "
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-model-cache\") pod \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") "
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-home" (OuterVolumeSpecName: "home") pod "dec046b7-d46b-49e6-8239-90e44ed50284" (UID: "dec046b7-d46b-49e6-8239-90e44ed50284"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "edaecc21-86b5-4e62-bcdd-90076b9ee8fd" (UID: "edaecc21-86b5-4e62-bcdd-90076b9ee8fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.933331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-dshm\") pod \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") "
Sep 14 10:50:24.933785 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933347    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-tmp-dir\") pod \"dec046b7-d46b-49e6-8239-90e44ed50284\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") "
Sep 14 10:50:24.933785 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec046b7-d46b-49e6-8239-90e44ed50284-tls-certs\") pod \"dec046b7-d46b-49e6-8239-90e44ed50284\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") "
Sep 14 10:50:24.933785 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-dshm\") pod \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") "
Sep 14 10:50:24.933785 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jfpv\" (UniqueName: \"kubernetes.io/projected/dec046b7-d46b-49e6-8239-90e44ed50284-kube-api-access-7jfpv\") pod \"dec046b7-d46b-49e6-8239-90e44ed50284\" (UID: \"dec046b7-d46b-49e6-8239-90e44ed50284\") "
Sep 14 10:50:24.933785 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tmp-dir\") pod \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\" (UID: \"edaecc21-86b5-4e62-bcdd-90076b9ee8fd\") "
Sep 14 10:50:24.933785 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933522    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tmp-dir\") pod \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\" (UID: \"8c2a5b6e-0294-4b34-9f8c-bc6219e04467\") "
Sep 14 10:50:24.934062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-model-cache" (OuterVolumeSpecName: "model-cache") pod "dec046b7-d46b-49e6-8239-90e44ed50284" (UID: "dec046b7-d46b-49e6-8239-90e44ed50284"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.934062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933938    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:24.934062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933959    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:24.934062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933970    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:24.934062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.933986    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:24.934297 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.934074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-home" (OuterVolumeSpecName: "home") pod "8c2a5b6e-0294-4b34-9f8c-bc6219e04467" (UID: "8c2a5b6e-0294-4b34-9f8c-bc6219e04467"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.934382 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.934346    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dec046b7-d46b-49e6-8239-90e44ed50284" (UID: "dec046b7-d46b-49e6-8239-90e44ed50284"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.934553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.934522    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edaecc21-86b5-4e62-bcdd-90076b9ee8fd" (UID: "edaecc21-86b5-4e62-bcdd-90076b9ee8fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.935916 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.935883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c2a5b6e-0294-4b34-9f8c-bc6219e04467" (UID: "8c2a5b6e-0294-4b34-9f8c-bc6219e04467"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:24.936121 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.936098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c2a5b6e-0294-4b34-9f8c-bc6219e04467" (UID: "8c2a5b6e-0294-4b34-9f8c-bc6219e04467"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.936324 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.936301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c2a5b6e-0294-4b34-9f8c-bc6219e04467" (UID: "8c2a5b6e-0294-4b34-9f8c-bc6219e04467"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.936401 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.936352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-dshm" (OuterVolumeSpecName: "dshm") pod "edaecc21-86b5-4e62-bcdd-90076b9ee8fd" (UID: "edaecc21-86b5-4e62-bcdd-90076b9ee8fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.936517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.936494    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-dshm" (OuterVolumeSpecName: "dshm") pod "dec046b7-d46b-49e6-8239-90e44ed50284" (UID: "dec046b7-d46b-49e6-8239-90e44ed50284"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.937053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.937019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-dshm" (OuterVolumeSpecName: "dshm") pod "8c2a5b6e-0294-4b34-9f8c-bc6219e04467" (UID: "8c2a5b6e-0294-4b34-9f8c-bc6219e04467"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:24.937301 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.937079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec046b7-d46b-49e6-8239-90e44ed50284-kube-api-access-7jfpv" (OuterVolumeSpecName: "kube-api-access-7jfpv") pod "dec046b7-d46b-49e6-8239-90e44ed50284" (UID: "dec046b7-d46b-49e6-8239-90e44ed50284"). InnerVolumeSpecName "kube-api-access-7jfpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:24.937540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.937516    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-kube-api-access-96cb6" (OuterVolumeSpecName: "kube-api-access-96cb6") pod "8c2a5b6e-0294-4b34-9f8c-bc6219e04467" (UID: "8c2a5b6e-0294-4b34-9f8c-bc6219e04467"). InnerVolumeSpecName "kube-api-access-96cb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:24.937721 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.937702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec046b7-d46b-49e6-8239-90e44ed50284-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dec046b7-d46b-49e6-8239-90e44ed50284" (UID: "dec046b7-d46b-49e6-8239-90e44ed50284"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:24.938151 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.938127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edaecc21-86b5-4e62-bcdd-90076b9ee8fd" (UID: "edaecc21-86b5-4e62-bcdd-90076b9ee8fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:24.938351 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:24.938331    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-kube-api-access-gfth8" (OuterVolumeSpecName: "kube-api-access-gfth8") pod "edaecc21-86b5-4e62-bcdd-90076b9ee8fd" (UID: "edaecc21-86b5-4e62-bcdd-90076b9ee8fd"). InnerVolumeSpecName "kube-api-access-gfth8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.034998    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035031    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035045    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035057    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dec046b7-d46b-49e6-8239-90e44ed50284-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035069    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035083    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfpv\" (UniqueName: \"kubernetes.io/projected/dec046b7-d46b-49e6-8239-90e44ed50284-kube-api-access-7jfpv\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035097    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035111    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035123    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035136    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035149    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96cb6\" (UniqueName: \"kubernetes.io/projected/8c2a5b6e-0294-4b34-9f8c-bc6219e04467-kube-api-access-96cb6\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035164    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035178    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/edaecc21-86b5-4e62-bcdd-90076b9ee8fd-kube-api-access-gfth8\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.035254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.035192    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dec046b7-d46b-49e6-8239-90e44ed50284-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:25.070724 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.070700    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:25.670875 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.670842    2563 generic.go:356] "Generic (PLEG): container finished" podID="dec046b7-d46b-49e6-8239-90e44ed50284" containerID="adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b" exitCode=0
Sep 14 10:50:25.671244 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.670890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"
Sep 14 10:50:25.671244 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.670920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn" event={"ID":"dec046b7-d46b-49e6-8239-90e44ed50284","Type":"ContainerDied","Data":"adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b"}
Sep 14 10:50:25.671244 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.670956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn" event={"ID":"dec046b7-d46b-49e6-8239-90e44ed50284","Type":"ContainerDied","Data":"4c8aaaceeae5d06307b0dbfd0d1eb1ff90fb7c0cf11592ed07f709315784af09"}
Sep 14 10:50:25.671244 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.670974    2563 scope.go:117] "RemoveContainer" containerID="adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b"
Sep 14 10:50:25.672828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.672591    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c2a5b6e-0294-4b34-9f8c-bc6219e04467" containerID="83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e" exitCode=0
Sep 14 10:50:25.672828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.672659    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"
Sep 14 10:50:25.672828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.672695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl" event={"ID":"8c2a5b6e-0294-4b34-9f8c-bc6219e04467","Type":"ContainerDied","Data":"83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e"}
Sep 14 10:50:25.672828 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.672739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl" event={"ID":"8c2a5b6e-0294-4b34-9f8c-bc6219e04467","Type":"ContainerDied","Data":"d2c528f5106f81ef3309a49d3771b6cbc1cb36cd5dffc53efa1717848b970a61"}
Sep 14 10:50:25.674551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.674501    2563 generic.go:356] "Generic (PLEG): container finished" podID="edaecc21-86b5-4e62-bcdd-90076b9ee8fd" containerID="24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0" exitCode=0
Sep 14 10:50:25.674722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.674535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p" event={"ID":"edaecc21-86b5-4e62-bcdd-90076b9ee8fd","Type":"ContainerDied","Data":"24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0"}
Sep 14 10:50:25.674722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.674632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p" event={"ID":"edaecc21-86b5-4e62-bcdd-90076b9ee8fd","Type":"ContainerDied","Data":"f4334b1820e107d3c8a5ed0a5768827aa1b5de6865d922e7b730a460439078f2"}
Sep 14 10:50:25.674722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.674676    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"
Sep 14 10:50:25.682711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.682690    2563 scope.go:117] "RemoveContainer" containerID="adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b"
Sep 14 10:50:25.683126 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:50:25.683081    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b\": container with ID starting with adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b not found: ID does not exist" containerID="adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b"
Sep 14 10:50:25.683222 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.683119    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b"} err="failed to get container status \"adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b\": rpc error: code = NotFound desc = could not find container \"adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b\": container with ID starting with adb29e88660ad8f67a21e4eae1085fc0c004c0b760838d5e5cf895f89641784b not found: ID does not exist"
Sep 14 10:50:25.683222 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.683145    2563 scope.go:117] "RemoveContainer" containerID="83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e"
Sep 14 10:50:25.692095 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.692073    2563 scope.go:117] "RemoveContainer" containerID="83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e"
Sep 14 10:50:25.692396 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:50:25.692377    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e\": container with ID starting with 83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e not found: ID does not exist" containerID="83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e"
Sep 14 10:50:25.692456 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.692403    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e"} err="failed to get container status \"83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e\": rpc error: code = NotFound desc = could not find container \"83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e\": container with ID starting with 83d779cc113be80ae9eebeb002e5f08d66016f096c592a76ac182e91d243b53e not found: ID does not exist"
Sep 14 10:50:25.692456 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.692421    2563 scope.go:117] "RemoveContainer" containerID="24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0"
Sep 14 10:50:25.700411 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.700385    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"]
Sep 14 10:50:25.702687 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.702667    2563 scope.go:117] "RemoveContainer" containerID="24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0"
Sep 14 10:50:25.702978 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:50:25.702962    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0\": container with ID starting with 24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0 not found: ID does not exist" containerID="24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0"
Sep 14 10:50:25.703042 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.702986    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0"} err="failed to get container status \"24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0\": rpc error: code = NotFound desc = could not find container \"24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0\": container with ID starting with 24617b3cd3846de0a8f883c33c2c09b5cc6f844bbde4d0dce790ffcf47a8b2f0 not found: ID does not exist"
Sep 14 10:50:25.710819 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.710793    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/distinct-2-w4-1910e9-kserve-64d88c9789-wjzdn"]
Sep 14 10:50:25.735738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.735710    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"]
Sep 14 10:50:25.744895 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.744869    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/simulator-w4-1910e9-kserve-77f95785f4-9rbbl"]
Sep 14 10:50:25.775327 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.775302    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"]
Sep 14 10:50:25.780732 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.780704    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1910e9/premium-w4-1910e9-kserve-74d75945b7-m574p"]
Sep 14 10:50:25.868155 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.868064    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"]
Sep 14 10:50:25.868314 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.868291    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph" podUID="89311ae2-10b5-497b-9ad8-037283ba1893" containerName="maas-api" containerID="cri-o://a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97" gracePeriod=30
Sep 14 10:50:25.907985 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.907955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:25.920018 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.919991    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2a5b6e-0294-4b34-9f8c-bc6219e04467" path="/var/lib/kubelet/pods/8c2a5b6e-0294-4b34-9f8c-bc6219e04467/volumes"
Sep 14 10:50:25.920390 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.920377    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec046b7-d46b-49e6-8239-90e44ed50284" path="/var/lib/kubelet/pods/dec046b7-d46b-49e6-8239-90e44ed50284/volumes"
Sep 14 10:50:25.920736 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:25.920723    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edaecc21-86b5-4e62-bcdd-90076b9ee8fd" path="/var/lib/kubelet/pods/edaecc21-86b5-4e62-bcdd-90076b9ee8fd/volumes"
Sep 14 10:50:26.006871 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.006841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:26.114775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.114750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:26.119522 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.119477    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:50:26.245929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.245894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89311ae2-10b5-497b-9ad8-037283ba1893-openshift-service-ca\") pod \"89311ae2-10b5-497b-9ad8-037283ba1893\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") "
Sep 14 10:50:26.245929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.245934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-metrics-certs\") pod \"89311ae2-10b5-497b-9ad8-037283ba1893\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") "
Sep 14 10:50:26.246125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.245996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-maas-api-tls\") pod \"89311ae2-10b5-497b-9ad8-037283ba1893\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") "
Sep 14 10:50:26.246125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.246024    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj2rs\" (UniqueName: \"kubernetes.io/projected/89311ae2-10b5-497b-9ad8-037283ba1893-kube-api-access-kj2rs\") pod \"89311ae2-10b5-497b-9ad8-037283ba1893\" (UID: \"89311ae2-10b5-497b-9ad8-037283ba1893\") "
Sep 14 10:50:26.246331 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.246301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89311ae2-10b5-497b-9ad8-037283ba1893-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89311ae2-10b5-497b-9ad8-037283ba1893" (UID: "89311ae2-10b5-497b-9ad8-037283ba1893"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:50:26.246493 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.246478    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89311ae2-10b5-497b-9ad8-037283ba1893-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:26.248253 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.248220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89311ae2-10b5-497b-9ad8-037283ba1893" (UID: "89311ae2-10b5-497b-9ad8-037283ba1893"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:26.248359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.248254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89311ae2-10b5-497b-9ad8-037283ba1893-kube-api-access-kj2rs" (OuterVolumeSpecName: "kube-api-access-kj2rs") pod "89311ae2-10b5-497b-9ad8-037283ba1893" (UID: "89311ae2-10b5-497b-9ad8-037283ba1893"). InnerVolumeSpecName "kube-api-access-kj2rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:26.248359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.248258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89311ae2-10b5-497b-9ad8-037283ba1893" (UID: "89311ae2-10b5-497b-9ad8-037283ba1893"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:26.278822 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.278793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:26.347792 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.347756    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:26.347792 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.347788    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89311ae2-10b5-497b-9ad8-037283ba1893-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:26.347792 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.347799    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2rs\" (UniqueName: \"kubernetes.io/projected/89311ae2-10b5-497b-9ad8-037283ba1893-kube-api-access-kj2rs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:26.556963 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.556926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:26.684063 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.684029    2563 generic.go:356] "Generic (PLEG): container finished" podID="89311ae2-10b5-497b-9ad8-037283ba1893" containerID="a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97" exitCode=0
Sep 14 10:50:26.684460 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.684084    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"
Sep 14 10:50:26.684460 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.684099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph" event={"ID":"89311ae2-10b5-497b-9ad8-037283ba1893","Type":"ContainerDied","Data":"a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97"}
Sep 14 10:50:26.684460 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.684144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph" event={"ID":"89311ae2-10b5-497b-9ad8-037283ba1893","Type":"ContainerDied","Data":"5b99a85f7420ba62650ec07d0dba320641cb7560b78df00485052447d9b81544"}
Sep 14 10:50:26.684460 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.684164    2563 scope.go:117] "RemoveContainer" containerID="a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97"
Sep 14 10:50:26.693246 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.693229    2563 scope.go:117] "RemoveContainer" containerID="a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97"
Sep 14 10:50:26.693513 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:50:26.693493    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97\": container with ID starting with a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97 not found: ID does not exist" containerID="a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97"
Sep 14 10:50:26.693561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.693520    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97"} err="failed to get container status \"a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97\": rpc error: code = NotFound desc = could not find container \"a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97\": container with ID starting with a245fac949f3ffec6b7e1f4b65ef68a909c4b112bfa33a0504a3d95d041d3b97 not found: ID does not exist"
Sep 14 10:50:26.733607 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.733587    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"]
Sep 14 10:50:26.752924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.752900    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea2835-598dcf8fbf-22kph"]
Sep 14 10:50:26.821864 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:26.821772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:27.565607 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:27.565561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:27.918747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:27.918708    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89311ae2-10b5-497b-9ad8-037283ba1893" path="/var/lib/kubelet/pods/89311ae2-10b5-497b-9ad8-037283ba1893/volumes"
Sep 14 10:50:28.323870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:28.323772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:29.166059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:29.166020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:29.768097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:29.768068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:29.869711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:29.869685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:30.264103 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:30.264068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:31.473380 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.473337    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c" podUID="093759ae-b088-4802-b36f-0d2c014b42e5" containerName="main" containerID="cri-o://207b394d2a53c10957909bdfae7006076edf8ac96f44f86111abf049058f2421" gracePeriod=45
Sep 14 10:50:31.713705 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.713679    2563 generic.go:356] "Generic (PLEG): container finished" podID="093759ae-b088-4802-b36f-0d2c014b42e5" containerID="207b394d2a53c10957909bdfae7006076edf8ac96f44f86111abf049058f2421" exitCode=0
Sep 14 10:50:31.713807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.713727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c" event={"ID":"093759ae-b088-4802-b36f-0d2c014b42e5","Type":"ContainerDied","Data":"207b394d2a53c10957909bdfae7006076edf8ac96f44f86111abf049058f2421"}
Sep 14 10:50:31.713807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.713755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c" event={"ID":"093759ae-b088-4802-b36f-0d2c014b42e5","Type":"ContainerDied","Data":"6fca66c02d0c6a4f2a8c07bf5015cf1932a189782f35a3d1eb443bb23250f2f7"}
Sep 14 10:50:31.713807 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.713767    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fca66c02d0c6a4f2a8c07bf5015cf1932a189782f35a3d1eb443bb23250f2f7"
Sep 14 10:50:31.722517 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.722496    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:50:31.796871 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.796805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093759ae-b088-4802-b36f-0d2c014b42e5-tls-certs\") pod \"093759ae-b088-4802-b36f-0d2c014b42e5\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") "
Sep 14 10:50:31.796871 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.796849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-home\") pod \"093759ae-b088-4802-b36f-0d2c014b42e5\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") "
Sep 14 10:50:31.797036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.796877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-dshm\") pod \"093759ae-b088-4802-b36f-0d2c014b42e5\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") "
Sep 14 10:50:31.797036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.796941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-tmp-dir\") pod \"093759ae-b088-4802-b36f-0d2c014b42e5\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") "
Sep 14 10:50:31.797036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.796968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjbsx\" (UniqueName: \"kubernetes.io/projected/093759ae-b088-4802-b36f-0d2c014b42e5-kube-api-access-jjbsx\") pod \"093759ae-b088-4802-b36f-0d2c014b42e5\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") "
Sep 14 10:50:31.797203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.797057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-home" (OuterVolumeSpecName: "home") pod "093759ae-b088-4802-b36f-0d2c014b42e5" (UID: "093759ae-b088-4802-b36f-0d2c014b42e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:31.797203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.797069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-model-cache\") pod \"093759ae-b088-4802-b36f-0d2c014b42e5\" (UID: \"093759ae-b088-4802-b36f-0d2c014b42e5\") "
Sep 14 10:50:31.797203 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.797167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "093759ae-b088-4802-b36f-0d2c014b42e5" (UID: "093759ae-b088-4802-b36f-0d2c014b42e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:31.797369 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.797261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "093759ae-b088-4802-b36f-0d2c014b42e5" (UID: "093759ae-b088-4802-b36f-0d2c014b42e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:31.797436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.797416    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:31.797436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.797429    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:31.797436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.797437    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:31.799202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.799181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093759ae-b088-4802-b36f-0d2c014b42e5-kube-api-access-jjbsx" (OuterVolumeSpecName: "kube-api-access-jjbsx") pod "093759ae-b088-4802-b36f-0d2c014b42e5" (UID: "093759ae-b088-4802-b36f-0d2c014b42e5"). InnerVolumeSpecName "kube-api-access-jjbsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:31.799202 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.799179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-dshm" (OuterVolumeSpecName: "dshm") pod "093759ae-b088-4802-b36f-0d2c014b42e5" (UID: "093759ae-b088-4802-b36f-0d2c014b42e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:50:31.799354 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.799239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093759ae-b088-4802-b36f-0d2c014b42e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "093759ae-b088-4802-b36f-0d2c014b42e5" (UID: "093759ae-b088-4802-b36f-0d2c014b42e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:31.898451 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.898414    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjbsx\" (UniqueName: \"kubernetes.io/projected/093759ae-b088-4802-b36f-0d2c014b42e5-kube-api-access-jjbsx\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:31.898451 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.898447    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093759ae-b088-4802-b36f-0d2c014b42e5-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:31.898451 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:31.898456    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093759ae-b088-4802-b36f-0d2c014b42e5-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:32.718910 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:32.718878    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"
Sep 14 10:50:32.744847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:32.744822    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"]
Sep 14 10:50:32.751458 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:32.751429    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/auth-test-model-7bdcda-kserve-5f975998d7-gq67c"]
Sep 14 10:50:33.328719 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:33.328694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:33.427811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:33.427780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:33.616728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:33.616596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:33.918321 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:33.918288    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093759ae-b088-4802-b36f-0d2c014b42e5" path="/var/lib/kubelet/pods/093759ae-b088-4802-b36f-0d2c014b42e5/volumes"
Sep 14 10:50:34.306389 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:34.306314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:35.251177 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:35.251144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:35.464428 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:35.464396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:50:36.147887 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:36.147853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:36.248324 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:36.248298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:37.035826 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:37.035789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:37.822917 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:37.822886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:38.716653 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:38.716619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:44.422419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.422379    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"]
Sep 14 10:50:44.422857 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.422600    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp" podUID="48fceceb-9b6d-4c90-b790-ee257746e4ca" containerName="maas-api" containerID="cri-o://3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51" gracePeriod=30
Sep 14 10:50:44.456769 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.456732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:44.658831 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.658805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:50:44.779052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.778968    2563 generic.go:356] "Generic (PLEG): container finished" podID="48fceceb-9b6d-4c90-b790-ee257746e4ca" containerID="3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51" exitCode=0
Sep 14 10:50:44.779052 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.779010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"
Sep 14 10:50:44.779231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.779047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp" event={"ID":"48fceceb-9b6d-4c90-b790-ee257746e4ca","Type":"ContainerDied","Data":"3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51"}
Sep 14 10:50:44.779231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.779096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp" event={"ID":"48fceceb-9b6d-4c90-b790-ee257746e4ca","Type":"ContainerDied","Data":"e86fe54c6e9f3474c2f3894a4d51fd09982a4c55d39dd3fc20bce15d751132ab"}
Sep 14 10:50:44.779231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.779118    2563 scope.go:117] "RemoveContainer" containerID="3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51"
Sep 14 10:50:44.787701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.787686    2563 scope.go:117] "RemoveContainer" containerID="3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51"
Sep 14 10:50:44.787948 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:50:44.787930    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51\": container with ID starting with 3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51 not found: ID does not exist" containerID="3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51"
Sep 14 10:50:44.788011 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.787959    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51"} err="failed to get container status \"3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51\": rpc error: code = NotFound desc = could not find container \"3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51\": container with ID starting with 3c3989c1252bcc718082ce333a5790bbceea9d0acfbfba6ad15901c19ff69d51 not found: ID does not exist"
Sep 14 10:50:44.812009 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.811966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls\") pod \"48fceceb-9b6d-4c90-b790-ee257746e4ca\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") "
Sep 14 10:50:44.812189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.812042    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wqht\" (UniqueName: \"kubernetes.io/projected/48fceceb-9b6d-4c90-b790-ee257746e4ca-kube-api-access-2wqht\") pod \"48fceceb-9b6d-4c90-b790-ee257746e4ca\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") "
Sep 14 10:50:44.812189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.812112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fceceb-9b6d-4c90-b790-ee257746e4ca-openshift-service-ca\") pod \"48fceceb-9b6d-4c90-b790-ee257746e4ca\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") "
Sep 14 10:50:44.812189 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.812149    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-metrics-certs\") pod \"48fceceb-9b6d-4c90-b790-ee257746e4ca\" (UID: \"48fceceb-9b6d-4c90-b790-ee257746e4ca\") "
Sep 14 10:50:44.812605 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.812556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fceceb-9b6d-4c90-b790-ee257746e4ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48fceceb-9b6d-4c90-b790-ee257746e4ca" (UID: "48fceceb-9b6d-4c90-b790-ee257746e4ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:50:44.814454 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.814422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "48fceceb-9b6d-4c90-b790-ee257746e4ca" (UID: "48fceceb-9b6d-4c90-b790-ee257746e4ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:44.814553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.814455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48fceceb-9b6d-4c90-b790-ee257746e4ca" (UID: "48fceceb-9b6d-4c90-b790-ee257746e4ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:50:44.814716 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.814696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fceceb-9b6d-4c90-b790-ee257746e4ca-kube-api-access-2wqht" (OuterVolumeSpecName: "kube-api-access-2wqht") pod "48fceceb-9b6d-4c90-b790-ee257746e4ca" (UID: "48fceceb-9b6d-4c90-b790-ee257746e4ca"). InnerVolumeSpecName "kube-api-access-2wqht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:50:44.913036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.912978    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqht\" (UniqueName: \"kubernetes.io/projected/48fceceb-9b6d-4c90-b790-ee257746e4ca-kube-api-access-2wqht\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:44.913036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.913031    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fceceb-9b6d-4c90-b790-ee257746e4ca-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:44.913036 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.913044    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:44.913328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:44.913056    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48fceceb-9b6d-4c90-b790-ee257746e4ca-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:50:45.123271 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:45.123234    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"]
Sep 14 10:50:45.124784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:45.124762    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3503d7-775469987c-b6pjp"]
Sep 14 10:50:45.918169 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:45.918133    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48fceceb-9b6d-4c90-b790-ee257746e4ca" path="/var/lib/kubelet/pods/48fceceb-9b6d-4c90-b790-ee257746e4ca/volumes"
Sep 14 10:50:46.160254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160216    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"]
Sep 14 10:50:46.160610 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160597    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160611    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160619    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fceceb-9b6d-4c90-b790-ee257746e4ca" containerName="maas-api"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fceceb-9b6d-4c90-b790-ee257746e4ca" containerName="maas-api"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160638    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160647    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9515cd-3937-48b8-b230-52a9d74fbb63" containerName="payload-processing"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9515cd-3937-48b8-b230-52a9d74fbb63" containerName="payload-processing"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160659    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec046b7-d46b-49e6-8239-90e44ed50284" containerName="main"
Sep 14 10:50:46.160663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160666    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec046b7-d46b-49e6-8239-90e44ed50284" containerName="main"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160796    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093759ae-b088-4802-b36f-0d2c014b42e5" containerName="main"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160802    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="093759ae-b088-4802-b36f-0d2c014b42e5" containerName="main"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160857    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89311ae2-10b5-497b-9ad8-037283ba1893" containerName="maas-api"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160863    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="89311ae2-10b5-497b-9ad8-037283ba1893" containerName="maas-api"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160870    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca2ed28-148f-43ca-8a79-84d5d17d8076" containerName="maas-api"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160875    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca2ed28-148f-43ca-8a79-84d5d17d8076" containerName="maas-api"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160883    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edaecc21-86b5-4e62-bcdd-90076b9ee8fd" containerName="main"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160888    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="edaecc21-86b5-4e62-bcdd-90076b9ee8fd" containerName="main"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160898    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2a5b6e-0294-4b34-9f8c-bc6219e04467" containerName="main"
Sep 14 10:50:46.161056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.160903    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2a5b6e-0294-4b34-9f8c-bc6219e04467" containerName="main"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161078    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2a5b6e-0294-4b34-9f8c-bc6219e04467" containerName="main"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161088    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fceceb-9b6d-4c90-b790-ee257746e4ca" containerName="maas-api"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161095    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="093759ae-b088-4802-b36f-0d2c014b42e5" containerName="main"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161102    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec046b7-d46b-49e6-8239-90e44ed50284" containerName="main"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161108    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca2ed28-148f-43ca-8a79-84d5d17d8076" containerName="maas-api"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161113    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="89311ae2-10b5-497b-9ad8-037283ba1893" containerName="maas-api"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161118    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161124    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ba1c2a-5736-423f-9cc7-b6ed4bddc20e" containerName="istio-proxy"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161131    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9515cd-3937-48b8-b230-52a9d74fbb63" containerName="payload-processing"
Sep 14 10:50:46.161459 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.161138    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="edaecc21-86b5-4e62-bcdd-90076b9ee8fd" containerName="main"
Sep 14 10:50:46.165682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.165656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.168922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.168864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1cbcd0\"/\"kube-root-ca.crt\""
Sep 14 10:50:46.168922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.168873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1cbcd0\"/\"sub-test-model-1cbcd0-kserve-self-signed-certs\""
Sep 14 10:50:46.168922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.168902    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1cbcd0\"/\"openshift-service-ca.crt\""
Sep 14 10:50:46.168922 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.168866    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1cbcd0\"/\"default-dockercfg-f9jpl\""
Sep 14 10:50:46.179645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.179616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"]
Sep 14 10:50:46.326268 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.326237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-tmp-dir\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.326442 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.326288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-dshm\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.326442 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.326307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1950743-bea1-4cc6-a2a5-879da695b688-tls-certs\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.326442 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.326323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56cpf\" (UniqueName: \"kubernetes.io/projected/c1950743-bea1-4cc6-a2a5-879da695b688-kube-api-access-56cpf\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.326442 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.326340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-home\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.326442 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.326382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-model-cache\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.415293 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.415270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:46.427677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.427619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-dshm\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.427677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.427647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1950743-bea1-4cc6-a2a5-879da695b688-tls-certs\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.427677 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.427664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56cpf\" (UniqueName: \"kubernetes.io/projected/c1950743-bea1-4cc6-a2a5-879da695b688-kube-api-access-56cpf\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.427867 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.427681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-home\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.427867 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.427712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-model-cache\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.427867 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.427778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-tmp-dir\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.428150 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.428130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-tmp-dir\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.428237 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.428156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-home\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.428237 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.428190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-model-cache\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.430125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.430105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-dshm\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.430313 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.430299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1950743-bea1-4cc6-a2a5-879da695b688-tls-certs\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.437316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.437293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56cpf\" (UniqueName: \"kubernetes.io/projected/c1950743-bea1-4cc6-a2a5-879da695b688-kube-api-access-56cpf\") pod \"sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") " pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.477358 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.477331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:46.664491 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.664451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:46.753997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.753974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"]
Sep 14 10:50:46.756096 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:50:46.756069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1950743_bea1_4cc6_a2a5_879da695b688.slice/crio-de6890bd95375752c0cfd911ed6f93d5fd37a5faf263dba456e87d209e694b01 WatchSource:0}: Error finding container de6890bd95375752c0cfd911ed6f93d5fd37a5faf263dba456e87d209e694b01: Status 404 returned error can't find the container with id de6890bd95375752c0cfd911ed6f93d5fd37a5faf263dba456e87d209e694b01
Sep 14 10:50:46.796701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.796667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v" event={"ID":"c1950743-bea1-4cc6-a2a5-879da695b688","Type":"ContainerStarted","Data":"de6890bd95375752c0cfd911ed6f93d5fd37a5faf263dba456e87d209e694b01"}
Sep 14 10:50:46.931340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:46.931308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:47.803907 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:47.803873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v" event={"ID":"c1950743-bea1-4cc6-a2a5-879da695b688","Type":"ContainerStarted","Data":"83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232"}
Sep 14 10:50:47.841537 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:47.841488    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v" podStartSLOduration=1.841474662 podStartE2EDuration="1.841474662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:47.840605597 +0000 UTC m=+1536.395793525" watchObservedRunningTime="2026-09-14 10:50:47.841474662 +0000 UTC m=+1536.396662588"
Sep 14 10:50:47.949783 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:47.949757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:48.739514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:48.739481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:51.172277 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:51.172231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:51.272512 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:51.272477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:51.462901 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:51.462813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:51.921543 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:51.921506    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:50:52.983751 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:52.983719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:53.790868 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:53.790840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:54.510467 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:54.510434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:54.723176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:54.723147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:55.351015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:55.350983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:50:56.477713 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:56.477683    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:56.478166 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:56.477729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:56.489731 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:56.489708    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:50:56.859391 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:50:56.859302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:51:02.390961 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:02.390921    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"]
Sep 14 10:51:02.397066 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:02.396973    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-558928-29823045-zffmk"]
Sep 14 10:51:02.407580 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:02.407526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:02.550118 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:02.549999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:02.649955 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:02.649919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:02.729259 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:02.729189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:03.032888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:03.032855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:03.305642 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:03.305531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:03.917920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:03.917885    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6ee48e6-82c5-45c4-a3e8-df83f596ce15" path="/var/lib/kubelet/pods/b6ee48e6-82c5-45c4-a3e8-df83f596ce15/volumes"
Sep 14 10:51:03.919696 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:03.919674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:05.316410 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:05.316378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:09.071959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:09.071919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:51:09.508858 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:09.508819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:09.605629 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:09.605549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:09.764531 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:09.764447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:12.111414 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:12.111385    2563 scope.go:117] "RemoveContainer" containerID="87d14f013a244e83ef0d5e83b035776699de97ced9c274db143447712018b2c8"
Sep 14 10:51:12.119187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:12.119168    2563 scope.go:117] "RemoveContainer" containerID="3c894c5c42da0ff111ef173b40ebf747c97f2b8fe421a20e73fad7a1742abb1a"
Sep 14 10:51:19.583665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.583633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"]
Sep 14 10:51:19.589249 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.589227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.592303 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.592274    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"kube-root-ca.crt\""
Sep 14 10:51:19.592418 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.592276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"sub-test-model-7bdcda-kserve-self-signed-certs\""
Sep 14 10:51:19.592418 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.592318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"default-dockercfg-b6p85\""
Sep 14 10:51:19.592418 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.592276    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"openshift-service-ca.crt\""
Sep 14 10:51:19.598868 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.598849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"]
Sep 14 10:51:19.668547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.668514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:19.700830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.700802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tls-certs\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.700947 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.700837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6fb\" (UniqueName: \"kubernetes.io/projected/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-kube-api-access-vd6fb\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.700947 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.700871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-dshm\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.700947 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.700928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-model-cache\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.701064 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.700962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tmp-dir\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.701064 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.701023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-home\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.801914 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.801876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-home\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.801931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tls-certs\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.801962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6fb\" (UniqueName: \"kubernetes.io/projected/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-kube-api-access-vd6fb\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802099 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.801998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-dshm\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.802140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-model-cache\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.802190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tmp-dir\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802387 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.802304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-home\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802450 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.802427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-model-cache\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.802490 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.802472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tmp-dir\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.804328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.804308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-dshm\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.804585 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.804553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tls-certs\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.809938 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.809915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6fb\" (UniqueName: \"kubernetes.io/projected/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-kube-api-access-vd6fb\") pod \"sub-test-model-7bdcda-kserve-c455f7485-rgbnj\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:19.900335 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:19.900313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:20.171678 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:20.171529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"]
Sep 14 10:51:20.174180 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:51:20.174151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45719cb7_caf7_4687_a6fc_2cb59f21ec4c.slice/crio-d58015c6c5d0cd3fe54f6d31e07cf57f044c6e90723c1746ee6773f92b7f1e36 WatchSource:0}: Error finding container d58015c6c5d0cd3fe54f6d31e07cf57f044c6e90723c1746ee6773f92b7f1e36: Status 404 returned error can't find the container with id d58015c6c5d0cd3fe54f6d31e07cf57f044c6e90723c1746ee6773f92b7f1e36
Sep 14 10:51:20.967323 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:20.967289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj" event={"ID":"45719cb7-caf7-4687-a6fc-2cb59f21ec4c","Type":"ContainerStarted","Data":"eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc"}
Sep 14 10:51:20.967685 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:20.967330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj" event={"ID":"45719cb7-caf7-4687-a6fc-2cb59f21ec4c","Type":"ContainerStarted","Data":"d58015c6c5d0cd3fe54f6d31e07cf57f044c6e90723c1746ee6773f92b7f1e36"}
Sep 14 10:51:21.007826 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:21.007763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj" podStartSLOduration=2.007746296 podStartE2EDuration="2.007746296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:51:21.006377726 +0000 UTC m=+1569.561565651" watchObservedRunningTime="2026-09-14 10:51:21.007746296 +0000 UTC m=+1569.562934236"
Sep 14 10:51:23.676456 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:23.676405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:51:29.901090 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:29.901056    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:29.901455 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:29.901105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:29.918439 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:29.918417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:30.019085 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:30.019063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:51:33.955393 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:33.955359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:40.372739 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:40.372698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:51:50.375763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:50.375730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:51:58.480017 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:58.479983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"]
Sep 14 10:51:58.506159 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:58.506121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:51:59.660608 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:59.660557    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"]
Sep 14 10:51:59.701864 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:51:59.701828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:52:01.577081 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:01.577044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:52:03.811520 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:03.811436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:52:13.069930 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.069896    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:52:13.498869 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.498828    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v" podUID="c1950743-bea1-4cc6-a2a5-879da695b688" containerName="main" containerID="cri-o://83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232" gracePeriod=45
Sep 14 10:52:13.749190 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.749135    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:52:13.844377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-dshm\") pod \"c1950743-bea1-4cc6-a2a5-879da695b688\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") "
Sep 14 10:52:13.844509 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56cpf\" (UniqueName: \"kubernetes.io/projected/c1950743-bea1-4cc6-a2a5-879da695b688-kube-api-access-56cpf\") pod \"c1950743-bea1-4cc6-a2a5-879da695b688\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") "
Sep 14 10:52:13.844509 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1950743-bea1-4cc6-a2a5-879da695b688-tls-certs\") pod \"c1950743-bea1-4cc6-a2a5-879da695b688\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") "
Sep 14 10:52:13.844509 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-home\") pod \"c1950743-bea1-4cc6-a2a5-879da695b688\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") "
Sep 14 10:52:13.844509 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844468    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-model-cache\") pod \"c1950743-bea1-4cc6-a2a5-879da695b688\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") "
Sep 14 10:52:13.844700 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844537    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-tmp-dir\") pod \"c1950743-bea1-4cc6-a2a5-879da695b688\" (UID: \"c1950743-bea1-4cc6-a2a5-879da695b688\") "
Sep 14 10:52:13.844761 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1950743-bea1-4cc6-a2a5-879da695b688" (UID: "c1950743-bea1-4cc6-a2a5-879da695b688"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:13.844761 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-home" (OuterVolumeSpecName: "home") pod "c1950743-bea1-4cc6-a2a5-879da695b688" (UID: "c1950743-bea1-4cc6-a2a5-879da695b688"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:13.844887 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1950743-bea1-4cc6-a2a5-879da695b688" (UID: "c1950743-bea1-4cc6-a2a5-879da695b688"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:13.844932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844882    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:13.844932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.844909    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:13.846773 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.846747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1950743-bea1-4cc6-a2a5-879da695b688-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1950743-bea1-4cc6-a2a5-879da695b688" (UID: "c1950743-bea1-4cc6-a2a5-879da695b688"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:52:13.846862 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.846755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-dshm" (OuterVolumeSpecName: "dshm") pod "c1950743-bea1-4cc6-a2a5-879da695b688" (UID: "c1950743-bea1-4cc6-a2a5-879da695b688"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:13.846862 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.846810    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1950743-bea1-4cc6-a2a5-879da695b688-kube-api-access-56cpf" (OuterVolumeSpecName: "kube-api-access-56cpf") pod "c1950743-bea1-4cc6-a2a5-879da695b688" (UID: "c1950743-bea1-4cc6-a2a5-879da695b688"). InnerVolumeSpecName "kube-api-access-56cpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:52:13.945599 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.945560    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:13.945599 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.945595    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1950743-bea1-4cc6-a2a5-879da695b688-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:13.945723 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.945604    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56cpf\" (UniqueName: \"kubernetes.io/projected/c1950743-bea1-4cc6-a2a5-879da695b688-kube-api-access-56cpf\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:13.945723 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:13.945613    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1950743-bea1-4cc6-a2a5-879da695b688-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:14.215185 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.215157    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1950743-bea1-4cc6-a2a5-879da695b688" containerID="83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232" exitCode=0
Sep 14 10:52:14.215515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.215194    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"
Sep 14 10:52:14.215515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.215211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v" event={"ID":"c1950743-bea1-4cc6-a2a5-879da695b688","Type":"ContainerDied","Data":"83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232"}
Sep 14 10:52:14.215515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.215247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v" event={"ID":"c1950743-bea1-4cc6-a2a5-879da695b688","Type":"ContainerDied","Data":"de6890bd95375752c0cfd911ed6f93d5fd37a5faf263dba456e87d209e694b01"}
Sep 14 10:52:14.215515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.215267    2563 scope.go:117] "RemoveContainer" containerID="83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232"
Sep 14 10:52:14.223929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.223911    2563 scope.go:117] "RemoveContainer" containerID="83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232"
Sep 14 10:52:14.224197 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:52:14.224178    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232\": container with ID starting with 83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232 not found: ID does not exist" containerID="83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232"
Sep 14 10:52:14.224250 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.224205    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232"} err="failed to get container status \"83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232\": rpc error: code = NotFound desc = could not find container \"83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232\": container with ID starting with 83efc62036f401c0fcc3e56f3c434e6394d7f8bd28f0427ecda73d2baa24a232 not found: ID does not exist"
Sep 14 10:52:14.232084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.232065    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"]
Sep 14 10:52:14.238286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.238266    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1cbcd0/sub-test-model-1cbcd0-kserve-8d6d548c7-7zt5v"]
Sep 14 10:52:14.679194 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.679157    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj" podUID="45719cb7-caf7-4687-a6fc-2cb59f21ec4c" containerName="main" containerID="cri-o://eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc" gracePeriod=45
Sep 14 10:52:14.918269 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:14.918246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:52:15.055690 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-model-cache\") pod \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") "
Sep 14 10:52:15.055690 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-dshm\") pod \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") "
Sep 14 10:52:15.055877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055707    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tmp-dir\") pod \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") "
Sep 14 10:52:15.055877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-home\") pod \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") "
Sep 14 10:52:15.055877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd6fb\" (UniqueName: \"kubernetes.io/projected/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-kube-api-access-vd6fb\") pod \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") "
Sep 14 10:52:15.055877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tls-certs\") pod \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\" (UID: \"45719cb7-caf7-4687-a6fc-2cb59f21ec4c\") "
Sep 14 10:52:15.056086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "45719cb7-caf7-4687-a6fc-2cb59f21ec4c" (UID: "45719cb7-caf7-4687-a6fc-2cb59f21ec4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:15.056086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.055942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45719cb7-caf7-4687-a6fc-2cb59f21ec4c" (UID: "45719cb7-caf7-4687-a6fc-2cb59f21ec4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:15.056086 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.056052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-home" (OuterVolumeSpecName: "home") pod "45719cb7-caf7-4687-a6fc-2cb59f21ec4c" (UID: "45719cb7-caf7-4687-a6fc-2cb59f21ec4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:15.056218 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.056096    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:15.056218 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.056110    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:15.057958 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.057929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-dshm" (OuterVolumeSpecName: "dshm") pod "45719cb7-caf7-4687-a6fc-2cb59f21ec4c" (UID: "45719cb7-caf7-4687-a6fc-2cb59f21ec4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:52:15.058055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.058003    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45719cb7-caf7-4687-a6fc-2cb59f21ec4c" (UID: "45719cb7-caf7-4687-a6fc-2cb59f21ec4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:52:15.058055 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.058024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-kube-api-access-vd6fb" (OuterVolumeSpecName: "kube-api-access-vd6fb") pod "45719cb7-caf7-4687-a6fc-2cb59f21ec4c" (UID: "45719cb7-caf7-4687-a6fc-2cb59f21ec4c"). InnerVolumeSpecName "kube-api-access-vd6fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:52:15.157034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.157010    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:15.157034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.157030    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:15.157034 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.157038    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:15.157193 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.157046    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6fb\" (UniqueName: \"kubernetes.io/projected/45719cb7-caf7-4687-a6fc-2cb59f21ec4c-kube-api-access-vd6fb\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:52:15.222395 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.222368    2563 generic.go:356] "Generic (PLEG): container finished" podID="45719cb7-caf7-4687-a6fc-2cb59f21ec4c" containerID="eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc" exitCode=0
Sep 14 10:52:15.222738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.222413    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"
Sep 14 10:52:15.222738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.222457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj" event={"ID":"45719cb7-caf7-4687-a6fc-2cb59f21ec4c","Type":"ContainerDied","Data":"eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc"}
Sep 14 10:52:15.222738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.222499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj" event={"ID":"45719cb7-caf7-4687-a6fc-2cb59f21ec4c","Type":"ContainerDied","Data":"d58015c6c5d0cd3fe54f6d31e07cf57f044c6e90723c1746ee6773f92b7f1e36"}
Sep 14 10:52:15.222738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.222521    2563 scope.go:117] "RemoveContainer" containerID="eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc"
Sep 14 10:52:15.231171 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.231151    2563 scope.go:117] "RemoveContainer" containerID="eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc"
Sep 14 10:52:15.231456 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:52:15.231430    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc\": container with ID starting with eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc not found: ID does not exist" containerID="eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc"
Sep 14 10:52:15.231508 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.231458    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc"} err="failed to get container status \"eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc\": rpc error: code = NotFound desc = could not find container \"eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc\": container with ID starting with eae0bbdbb3dbcabd4251a2478ce08653aa25c0cb520bc98e45e0e0c413444efc not found: ID does not exist"
Sep 14 10:52:15.246779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.246756    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"]
Sep 14 10:52:15.249211 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.249192    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/sub-test-model-7bdcda-kserve-c455f7485-rgbnj"]
Sep 14 10:52:15.923301 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.923263    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45719cb7-caf7-4687-a6fc-2cb59f21ec4c" path="/var/lib/kubelet/pods/45719cb7-caf7-4687-a6fc-2cb59f21ec4c/volumes"
Sep 14 10:52:15.923662 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:15.923647    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1950743-bea1-4cc6-a2a5-879da695b688" path="/var/lib/kubelet/pods/c1950743-bea1-4cc6-a2a5-879da695b688/volumes"
Sep 14 10:52:29.975299 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.975267    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"]
Sep 14 10:52:29.975784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.975719    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1950743-bea1-4cc6-a2a5-879da695b688" containerName="main"
Sep 14 10:52:29.975784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.975736    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1950743-bea1-4cc6-a2a5-879da695b688" containerName="main"
Sep 14 10:52:29.975784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.975758    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45719cb7-caf7-4687-a6fc-2cb59f21ec4c" containerName="main"
Sep 14 10:52:29.975784 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.975766    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="45719cb7-caf7-4687-a6fc-2cb59f21ec4c" containerName="main"
Sep 14 10:52:29.976018 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.975850    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="45719cb7-caf7-4687-a6fc-2cb59f21ec4c" containerName="main"
Sep 14 10:52:29.976018 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.975862    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1950743-bea1-4cc6-a2a5-879da695b688" containerName="main"
Sep 14 10:52:29.980195 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.980171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:29.982358 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.982336    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"kube-root-ca.crt\""
Sep 14 10:52:29.983220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.983201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"rate-test-model-7bdcda-kserve-self-signed-certs\""
Sep 14 10:52:29.983333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.983205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"default-dockercfg-b6p85\""
Sep 14 10:52:29.983333 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.983267    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7bdcda\"/\"openshift-service-ca.crt\""
Sep 14 10:52:29.991614 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:29.991590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"]
Sep 14 10:52:30.022081 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.022057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:52:30.074639 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.074611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-dshm\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.074763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.074646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878b304d-51f7-4dba-82c0-57cd8720fd7c-tls-certs\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.074763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.074732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-tmp-dir\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.074876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.074764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-home\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.074876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.074806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lm2\" (UniqueName: \"kubernetes.io/projected/878b304d-51f7-4dba-82c0-57cd8720fd7c-kube-api-access-b7lm2\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.074876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.074855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-model-cache\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.176282 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.176255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lm2\" (UniqueName: \"kubernetes.io/projected/878b304d-51f7-4dba-82c0-57cd8720fd7c-kube-api-access-b7lm2\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.176401 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.176298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-model-cache\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.176561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.176537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-dshm\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.176681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.176605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878b304d-51f7-4dba-82c0-57cd8720fd7c-tls-certs\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.176745 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.176732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-model-cache\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.176802 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.176757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-tmp-dir\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.176802 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.176795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-home\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.177056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.177036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-tmp-dir\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.177139 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.177097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-home\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.179285 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.179263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-dshm\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.179669 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.179649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878b304d-51f7-4dba-82c0-57cd8720fd7c-tls-certs\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.185826 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.185809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lm2\" (UniqueName: \"kubernetes.io/projected/878b304d-51f7-4dba-82c0-57cd8720fd7c-kube-api-access-b7lm2\") pod \"rate-test-model-7bdcda-kserve-5444cb98c5-66wrd\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") " pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.289746 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.289683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:30.570187 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:30.570108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"]
Sep 14 10:52:30.573122 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:52:30.573095    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod878b304d_51f7_4dba_82c0_57cd8720fd7c.slice/crio-4fa5c68b0093f5a041c50b43828dd88bbf02693c5ed50719ae45390f7b8fad5c WatchSource:0}: Error finding container 4fa5c68b0093f5a041c50b43828dd88bbf02693c5ed50719ae45390f7b8fad5c: Status 404 returned error can't find the container with id 4fa5c68b0093f5a041c50b43828dd88bbf02693c5ed50719ae45390f7b8fad5c
Sep 14 10:52:31.301063 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:31.301026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd" event={"ID":"878b304d-51f7-4dba-82c0-57cd8720fd7c","Type":"ContainerStarted","Data":"2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7"}
Sep 14 10:52:31.301486 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:31.301078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd" event={"ID":"878b304d-51f7-4dba-82c0-57cd8720fd7c","Type":"ContainerStarted","Data":"4fa5c68b0093f5a041c50b43828dd88bbf02693c5ed50719ae45390f7b8fad5c"}
Sep 14 10:52:31.321425 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:31.321378    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd" podStartSLOduration=2.321354007 podStartE2EDuration="2.321354007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:52:31.320227447 +0000 UTC m=+1639.875415372" watchObservedRunningTime="2026-09-14 10:52:31.321354007 +0000 UTC m=+1639.876541932"
Sep 14 10:52:33.779722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:33.779689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:52:40.290149 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:40.290114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:40.290612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:40.290192    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:40.301796 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:40.301775    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:40.356898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:40.356873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:52:52.967479 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:52:52.967439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:53:02.476419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:02.476335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:53:07.222397 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:07.222367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:08.129440 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:08.129408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:09.100685 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:09.100647    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"]
Sep 14 10:53:09.123551 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:09.123518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:09.548413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:09.548375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:09.863267 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:09.863183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:10.206046 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:10.206008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:11.945340 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:11.945285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:12.172543 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:12.172513    2563 scope.go:117] "RemoveContainer" containerID="207b394d2a53c10957909bdfae7006076edf8ac96f44f86111abf049058f2421"
Sep 14 10:53:12.643312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:12.643271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:12.739256 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:12.739201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:12.840888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:12.840849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:13.637254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:13.637225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:13.689025 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:13.688988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:53:14.396898 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.396863    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"]
Sep 14 10:53:14.400410 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.400387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.402260 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.402239    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-689fac-serving-cert\""
Sep 14 10:53:14.412487 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.412459    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"]
Sep 14 10:53:14.446050 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.446012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:14.544671 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.544640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-maas-api-tls\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.544841 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.544775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vxw\" (UniqueName: \"kubernetes.io/projected/3c757dfb-a14d-4903-a48a-db2674ff7698-kube-api-access-n2vxw\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.544888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.544836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c757dfb-a14d-4903-a48a-db2674ff7698-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.544888 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.544873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-metrics-certs\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.553862 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.553836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:14.646291 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.646258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c757dfb-a14d-4903-a48a-db2674ff7698-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.646710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.646304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-metrics-certs\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.646710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.646321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-maas-api-tls\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.646710 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.646398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vxw\" (UniqueName: \"kubernetes.io/projected/3c757dfb-a14d-4903-a48a-db2674ff7698-kube-api-access-n2vxw\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.647048 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.647003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c757dfb-a14d-4903-a48a-db2674ff7698-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.648779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.648754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-maas-api-tls\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.648929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.648905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-metrics-certs\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.653858 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.653834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vxw\" (UniqueName: \"kubernetes.io/projected/3c757dfb-a14d-4903-a48a-db2674ff7698-kube-api-access-n2vxw\") pod \"maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:14.673834 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.673814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:14.710473 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:14.710450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:15.012846 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:15.012822    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"]
Sep 14 10:53:15.014509 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:53:15.014485    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c757dfb_a14d_4903_a48a_db2674ff7698.slice/crio-e3d12229240655141401ce0608fec88ab4899dceb165732b54744ff9e0f46674 WatchSource:0}: Error finding container e3d12229240655141401ce0608fec88ab4899dceb165732b54744ff9e0f46674: Status 404 returned error can't find the container with id e3d12229240655141401ce0608fec88ab4899dceb165732b54744ff9e0f46674
Sep 14 10:53:15.148876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:15.148846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:15.293288 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:15.293205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:15.524013 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:15.523981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r" event={"ID":"3c757dfb-a14d-4903-a48a-db2674ff7698","Type":"ContainerStarted","Data":"e3d12229240655141401ce0608fec88ab4899dceb165732b54744ff9e0f46674"}
Sep 14 10:53:16.531057 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:16.531017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r" event={"ID":"3c757dfb-a14d-4903-a48a-db2674ff7698","Type":"ContainerStarted","Data":"31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41"}
Sep 14 10:53:16.531432 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:16.531202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:16.553849 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:16.553804    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r" podStartSLOduration=2.014208081 podStartE2EDuration="2.553790896s" totalImagesPullingTime="539.582815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:53:16.55147037 +0000 UTC m=+1685.106658296" watchObservedRunningTime="2026-09-14 10:53:16.553790896 +0000 UTC m=+1685.108978851"
Sep 14 10:53:22.541768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:22.541740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:53:24.118964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.118927    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd" podUID="878b304d-51f7-4dba-82c0-57cd8720fd7c" containerName="main" containerID="cri-o://2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7" gracePeriod=45
Sep 14 10:53:24.360144 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.360114    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:53:24.431805 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.431765    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-dshm\") pod \"878b304d-51f7-4dba-82c0-57cd8720fd7c\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") "
Sep 14 10:53:24.431997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.431812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878b304d-51f7-4dba-82c0-57cd8720fd7c-tls-certs\") pod \"878b304d-51f7-4dba-82c0-57cd8720fd7c\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") "
Sep 14 10:53:24.431997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.431840    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-model-cache\") pod \"878b304d-51f7-4dba-82c0-57cd8720fd7c\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") "
Sep 14 10:53:24.431997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.431924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7lm2\" (UniqueName: \"kubernetes.io/projected/878b304d-51f7-4dba-82c0-57cd8720fd7c-kube-api-access-b7lm2\") pod \"878b304d-51f7-4dba-82c0-57cd8720fd7c\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") "
Sep 14 10:53:24.431997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.431984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-tmp-dir\") pod \"878b304d-51f7-4dba-82c0-57cd8720fd7c\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") "
Sep 14 10:53:24.432204 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.432024    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-home\") pod \"878b304d-51f7-4dba-82c0-57cd8720fd7c\" (UID: \"878b304d-51f7-4dba-82c0-57cd8720fd7c\") "
Sep 14 10:53:24.432204 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.432146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "878b304d-51f7-4dba-82c0-57cd8720fd7c" (UID: "878b304d-51f7-4dba-82c0-57cd8720fd7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:53:24.432316 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.432274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "878b304d-51f7-4dba-82c0-57cd8720fd7c" (UID: "878b304d-51f7-4dba-82c0-57cd8720fd7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:53:24.432368 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.432313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-home" (OuterVolumeSpecName: "home") pod "878b304d-51f7-4dba-82c0-57cd8720fd7c" (UID: "878b304d-51f7-4dba-82c0-57cd8720fd7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:53:24.432368 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.432336    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:53:24.433928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.433907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-dshm" (OuterVolumeSpecName: "dshm") pod "878b304d-51f7-4dba-82c0-57cd8720fd7c" (UID: "878b304d-51f7-4dba-82c0-57cd8720fd7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:53:24.434104 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.434086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878b304d-51f7-4dba-82c0-57cd8720fd7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "878b304d-51f7-4dba-82c0-57cd8720fd7c" (UID: "878b304d-51f7-4dba-82c0-57cd8720fd7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:53:24.434170 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.434156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878b304d-51f7-4dba-82c0-57cd8720fd7c-kube-api-access-b7lm2" (OuterVolumeSpecName: "kube-api-access-b7lm2") pod "878b304d-51f7-4dba-82c0-57cd8720fd7c" (UID: "878b304d-51f7-4dba-82c0-57cd8720fd7c"). InnerVolumeSpecName "kube-api-access-b7lm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:53:24.532941 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.532905    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:53:24.532941 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.532935    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:53:24.532941 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.532943    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/878b304d-51f7-4dba-82c0-57cd8720fd7c-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:53:24.532941 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.532951    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/878b304d-51f7-4dba-82c0-57cd8720fd7c-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:53:24.533188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.532959    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7lm2\" (UniqueName: \"kubernetes.io/projected/878b304d-51f7-4dba-82c0-57cd8720fd7c-kube-api-access-b7lm2\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:53:24.570782 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.570746    2563 generic.go:356] "Generic (PLEG): container finished" podID="878b304d-51f7-4dba-82c0-57cd8720fd7c" containerID="2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7" exitCode=0
Sep 14 10:53:24.570916 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.570794    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"
Sep 14 10:53:24.570916 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.570837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd" event={"ID":"878b304d-51f7-4dba-82c0-57cd8720fd7c","Type":"ContainerDied","Data":"2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7"}
Sep 14 10:53:24.570916 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.570881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd" event={"ID":"878b304d-51f7-4dba-82c0-57cd8720fd7c","Type":"ContainerDied","Data":"4fa5c68b0093f5a041c50b43828dd88bbf02693c5ed50719ae45390f7b8fad5c"}
Sep 14 10:53:24.570916 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.570899    2563 scope.go:117] "RemoveContainer" containerID="2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7"
Sep 14 10:53:24.579637 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.579621    2563 scope.go:117] "RemoveContainer" containerID="2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7"
Sep 14 10:53:24.579911 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:53:24.579893    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7\": container with ID starting with 2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7 not found: ID does not exist" containerID="2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7"
Sep 14 10:53:24.579957 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.579918    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7"} err="failed to get container status \"2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7\": rpc error: code = NotFound desc = could not find container \"2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7\": container with ID starting with 2eddeef4dd551fcfd7a5529ef592ce760c8c8281b43ca34226f214655c99e6e7 not found: ID does not exist"
Sep 14 10:53:24.593647 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.593624    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"]
Sep 14 10:53:24.600265 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.600245    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7bdcda/rate-test-model-7bdcda-kserve-5444cb98c5-66wrd"]
Sep 14 10:53:24.683378 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:24.683295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:53:25.919355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:25.919321    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="878b304d-51f7-4dba-82c0-57cd8720fd7c" path="/var/lib/kubelet/pods/878b304d-51f7-4dba-82c0-57cd8720fd7c/volumes"
Sep 14 10:53:36.791361 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:36.791274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:37.490438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:37.490407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:37.593417 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:37.593368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:38.261005 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.260972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:38.361663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.361629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:38.545380 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.545288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"]
Sep 14 10:53:38.545735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.545720    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="878b304d-51f7-4dba-82c0-57cd8720fd7c" containerName="main"
Sep 14 10:53:38.545778 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.545739    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="878b304d-51f7-4dba-82c0-57cd8720fd7c" containerName="main"
Sep 14 10:53:38.545826 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.545810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="878b304d-51f7-4dba-82c0-57cd8720fd7c" containerName="main"
Sep 14 10:53:38.550547 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.550528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.553870 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.553853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7406d4-openshift-default-dockercfg-4knrl\""
Sep 14 10:53:38.556875 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.556849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-token\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.556897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istiod-ca-cert\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.556953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-envoy\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.556990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsl5g\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-kube-api-access-gsl5g\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.557031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-socket\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.557064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-credential-socket\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.557089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-data\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.557129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-certs\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.557492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.557161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-podinfo\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.571672 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.571647    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"]
Sep 14 10:53:38.597806 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.597785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:38.657856 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.657832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-token\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.657868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istiod-ca-cert\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.657892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-envoy\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.657916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsl5g\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-kube-api-access-gsl5g\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.657940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-socket\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.657958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-credential-socket\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-data\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-certs\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658254 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-podinfo\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658401 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-credential-socket\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658401 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-socket\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-data\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-certs\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.658725 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.658592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istiod-ca-cert\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.660398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.660369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-envoy\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.660705 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.660686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-podinfo\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.665978 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.665960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-token\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.666358 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.666339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsl5g\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-kube-api-access-gsl5g\") pod \"e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") " pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:38.698767 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.698721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:38.860600 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:38.860495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:39.140881 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.140840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"]
Sep 14 10:53:39.143845 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:53:39.143812    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaeb9c1e_5a22_48c0_94bb_4c0525559bc4.slice/crio-c60c75f7057ded0cb7ecfd92937b984a58293587431d2c4197fa5e5ada7f1c4e WatchSource:0}: Error finding container c60c75f7057ded0cb7ecfd92937b984a58293587431d2c4197fa5e5ada7f1c4e: Status 404 returned error can't find the container with id c60c75f7057ded0cb7ecfd92937b984a58293587431d2c4197fa5e5ada7f1c4e
Sep 14 10:53:39.146060 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.146031    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:53:39.146141 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.146095    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:53:39.146141 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.146126    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:53:39.186475 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.186449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:39.482622 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.482524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:39.651496 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.651455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" event={"ID":"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4","Type":"ContainerStarted","Data":"aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096"}
Sep 14 10:53:39.651682 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.651513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" event={"ID":"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4","Type":"ContainerStarted","Data":"c60c75f7057ded0cb7ecfd92937b984a58293587431d2c4197fa5e5ada7f1c4e"}
Sep 14 10:53:39.675321 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.675270    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podStartSLOduration=1.6752549939999999 podStartE2EDuration="1.675254994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:53:39.673732375 +0000 UTC m=+1708.228920326" watchObservedRunningTime="2026-09-14 10:53:39.675254994 +0000 UTC m=+1708.230442918"
Sep 14 10:53:39.861063 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:39.860982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:40.865565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:40.865532    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:40.972377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:40.972347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:41.610311 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:41.610274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:41.660701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:41.660672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:41.661526 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:41.661503    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:41.760925 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:41.760893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:41.964406 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:41.964375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:42.738820 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:42.738790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:43.572433 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:43.572399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:44.364438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:44.364407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:44.451535 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:44.451502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:44.683892 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:44.683863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:45.147563 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.147534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:45.246385 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.246355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:45.300891 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.300859    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"]
Sep 14 10:53:45.304071 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.304053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.306102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.306078    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7406d4\""
Sep 14 10:53:45.306231 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.306108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7406d4-dockercfg-spz9p\""
Sep 14 10:53:45.313096 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.313070    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"]
Sep 14 10:53:45.316305 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.316279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnn9q\" (UniqueName: \"kubernetes.io/projected/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-kube-api-access-cnn9q\") pod \"payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.316421 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.316403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.417362 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.417281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnn9q\" (UniqueName: \"kubernetes.io/projected/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-kube-api-access-cnn9q\") pod \"payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.417362 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.417357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.418135 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.418096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.432405 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.432382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnn9q\" (UniqueName: \"kubernetes.io/projected/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-kube-api-access-cnn9q\") pod \"payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.615809 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.615777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:45.887548 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:45.887420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"]
Sep 14 10:53:45.890628 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:53:45.890599    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fb1c4b_b678_40a1_8202_64ee6efc8f6a.slice/crio-b488064e5f59d533a31e30e93d3866fc958660f86346fcc7d0576242ca44073d WatchSource:0}: Error finding container b488064e5f59d533a31e30e93d3866fc958660f86346fcc7d0576242ca44073d: Status 404 returned error can't find the container with id b488064e5f59d533a31e30e93d3866fc958660f86346fcc7d0576242ca44073d
Sep 14 10:53:46.691024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:46.690984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v" event={"ID":"02fb1c4b-b678-40a1-8202-64ee6efc8f6a","Type":"ContainerStarted","Data":"10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54"}
Sep 14 10:53:46.691199 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:46.691087    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:46.691199 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:46.691114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v" event={"ID":"02fb1c4b-b678-40a1-8202-64ee6efc8f6a","Type":"ContainerStarted","Data":"b488064e5f59d533a31e30e93d3866fc958660f86346fcc7d0576242ca44073d"}
Sep 14 10:53:46.712056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:46.712002    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v" podStartSLOduration=1.711987363 podStartE2EDuration="1.711987363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:53:46.709601196 +0000 UTC m=+1715.264789119" watchObservedRunningTime="2026-09-14 10:53:46.711987363 +0000 UTC m=+1715.267175292"
Sep 14 10:53:47.397403 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:47.397361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:53:49.199139 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:49.199097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:53:57.699192 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:57.699156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:53:57.702689 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:57.702665    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 14 10:53:57.702797 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:57.702712    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 14 10:53:58.766344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:58.766307    2563 generic.go:356] "Generic (PLEG): container finished" podID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerID="aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096" exitCode=0
Sep 14 10:53:58.766775 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:58.766378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" event={"ID":"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4","Type":"ContainerDied","Data":"aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096"}
Sep 14 10:53:58.766817 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:58.766788    2563 scope.go:117] "RemoveContainer" containerID="aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096"
Sep 14 10:53:58.767929 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:58.767905    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:53:58.768006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:58.767970    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:53:58.768043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:58.768002    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:53:59.773219 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:59.773180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" event={"ID":"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4","Type":"ContainerStarted","Data":"46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9"}
Sep 14 10:53:59.861075 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:59.861046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:53:59.861281 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:59.861259    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:53:59.861330 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:53:59.861309    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:54:00.861170 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:00.861137    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:54:00.861538 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:00.861192    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:54:01.860949 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:01.860908    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:54:01.861148 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:01.860982    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:54:02.861420 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:02.861384    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:54:02.861825 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:02.861462    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:54:03.861647 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:03.861612    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Sep 14 10:54:03.862018 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:03.861680    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Sep 14 10:54:04.864364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:04.864329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:54:04.864739 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:04.864665    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:54:04.865364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:04.865344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:54:05.593676 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:05.593644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:07.004318 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:07.004273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:07.720257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:07.720222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:08.537205 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:08.537176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:09.491102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:09.491068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:10.175088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:10.175064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:11.468271 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:11.468240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:11.672928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:11.672892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:22.741793 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.741761    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"]
Sep 14 10:54:22.744375 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.744357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.747056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.747033    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-689fac\"/\"default-dockercfg-pxtvg\""
Sep 14 10:54:22.747174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.747068    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-689fac\"/\"ipp-route-689fac-kserve-self-signed-certs\""
Sep 14 10:54:22.747174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.747070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-689fac\"/\"openshift-service-ca.crt\""
Sep 14 10:54:22.747174 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.747130    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-689fac\"/\"kube-root-ca.crt\""
Sep 14 10:54:22.757631 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.757609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:22.757728 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.757647    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"]
Sep 14 10:54:22.852062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.852028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwc8r\" (UniqueName: \"kubernetes.io/projected/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-kube-api-access-kwc8r\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.852062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.852066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-home\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.852274 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.852125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-dshm\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.852274 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.852160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-model-cache\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.852274 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.852209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tls-certs\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.852274 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.852236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tmp-dir\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.953422 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwc8r\" (UniqueName: \"kubernetes.io/projected/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-kube-api-access-kwc8r\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.953623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-home\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.953623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-dshm\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.953711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-model-cache\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.953711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tls-certs\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.953711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tmp-dir\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.953865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-home\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.954007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.953986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-model-cache\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.954043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.954021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tmp-dir\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.955825 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.955808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-dshm\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.956087 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.956071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tls-certs\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:22.962083 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:22.962058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwc8r\" (UniqueName: \"kubernetes.io/projected/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-kube-api-access-kwc8r\") pod \"ipp-route-689fac-kserve-8f79f5c9f-b7s7w\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") " pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:23.054007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:23.053946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:23.328154 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:23.328128    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"]
Sep 14 10:54:23.331052 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:54:23.331012    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a35ef6_10f7_4185_9f85_bbdd381d2c2e.slice/crio-9c21242ac26b81c17d7da16533c3b1992d683a701a81dc34143e432b7e7ed8a1 WatchSource:0}: Error finding container 9c21242ac26b81c17d7da16533c3b1992d683a701a81dc34143e432b7e7ed8a1: Status 404 returned error can't find the container with id 9c21242ac26b81c17d7da16533c3b1992d683a701a81dc34143e432b7e7ed8a1
Sep 14 10:54:23.890358 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:23.890326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w" event={"ID":"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e","Type":"ContainerStarted","Data":"1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c"}
Sep 14 10:54:23.890358 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:23.890366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w" event={"ID":"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e","Type":"ContainerStarted","Data":"9c21242ac26b81c17d7da16533c3b1992d683a701a81dc34143e432b7e7ed8a1"}
Sep 14 10:54:23.930604 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:23.930534    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w" podStartSLOduration=1.930521038 podStartE2EDuration="1.930521038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:54:23.928223697 +0000 UTC m=+1752.483411622" watchObservedRunningTime="2026-09-14 10:54:23.930521038 +0000 UTC m=+1752.485708964"
Sep 14 10:54:26.871656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:26.871621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:54:33.054294 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:33.054258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:33.054294 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:33.054304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:33.066590 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:33.066554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:33.951084 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:33.951052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:54:39.733928 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:39.733896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:41.372025 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:41.371983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:54:45.875444 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:45.875405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:47.967445 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:47.967409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:48.468712 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.468685    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"]
Sep 14 10:54:48.471359 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.471343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.473755 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.473735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-7599d7-openshift-default-dockercfg-6wxml\""
Sep 14 10:54:48.489623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.489560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"]
Sep 14 10:54:48.524225 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.524202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:48.572769 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.572731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-podinfo\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.572769 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.572773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8r8f\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-kube-api-access-p8r8f\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.573053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.572802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1094696e-6623-4bc0-8ac2-22f1683baab1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.573053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.572884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-socket\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.573053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.572905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-token\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.573053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.572934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-certs\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.573243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.573048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-data\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.573243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.573106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-envoy\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.573243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.573134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-credential-socket\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674508 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-podinfo\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8r8f\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-kube-api-access-p8r8f\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1094696e-6623-4bc0-8ac2-22f1683baab1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674666 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-socket\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674838 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-token\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674838 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-certs\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674939 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-data\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674993 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-envoy\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.674993 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-credential-socket\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.675096 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.674991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-socket\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.675096 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.675053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-certs\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.675271 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.675250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-data\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.675344 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.675328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1094696e-6623-4bc0-8ac2-22f1683baab1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.675563 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.675543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-credential-socket\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.677067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.677047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-envoy\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.677270 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.677250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-podinfo\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.683997 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.683973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-token\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.684258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.684241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8r8f\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-kube-api-access-p8r8f\") pod \"e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") " pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.782097 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.782045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:48.835539 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:48.835232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:49.061823 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:49.061799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"]
Sep 14 10:54:49.063646 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:54:49.063616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1094696e_6623_4bc0_8ac2_22f1683baab1.slice/crio-87f83eaec89431802595a5a67526f4b74029ffde7c570884a321425c401c8e1a WatchSource:0}: Error finding container 87f83eaec89431802595a5a67526f4b74029ffde7c570884a321425c401c8e1a: Status 404 returned error can't find the container with id 87f83eaec89431802595a5a67526f4b74029ffde7c570884a321425c401c8e1a
Sep 14 10:54:49.065832 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:49.065801    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:54:49.065943 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:49.065872    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:54:49.065943 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:49.065916    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:54:50.014699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:50.014660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" event={"ID":"1094696e-6623-4bc0-8ac2-22f1683baab1","Type":"ContainerStarted","Data":"37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108"}
Sep 14 10:54:50.014868 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:50.014713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" event={"ID":"1094696e-6623-4bc0-8ac2-22f1683baab1","Type":"ContainerStarted","Data":"87f83eaec89431802595a5a67526f4b74029ffde7c570884a321425c401c8e1a"}
Sep 14 10:54:50.037652 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:50.037602    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" podStartSLOduration=2.037563121 podStartE2EDuration="2.037563121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:54:50.035781012 +0000 UTC m=+1778.590968949" watchObservedRunningTime="2026-09-14 10:54:50.037563121 +0000 UTC m=+1778.592751045"
Sep 14 10:54:50.782560 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:50.782530    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:50.787395 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:50.787374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:51.019763 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:51.019730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:51.020701 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:51.020680    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:54:51.297127 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:51.297086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:51.935838 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:51.935804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:52.028398 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:52.028365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:52.314854 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:52.314749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:53.339373 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:53.339317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:54.175102 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:54.175015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:54.892089 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:54.892053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:55.744432 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:55.744373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:55.840731 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:55.840694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:56.318791 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.318749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:56.491749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.491716    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"]
Sep 14 10:54:56.494391 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.494369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:56.501468 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.501445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7599d7-dockercfg-p8d5z\""
Sep 14 10:54:56.501737 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.501718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7599d7\""
Sep 14 10:54:56.516844 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.516824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"]
Sep 14 10:54:56.523323 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.523305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:54:56.648238 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.648202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c39b306-58f3-4cab-848b-48f674d404a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:56.648412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.648256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52t2\" (UniqueName: \"kubernetes.io/projected/7c39b306-58f3-4cab-848b-48f674d404a4-kube-api-access-r52t2\") pod \"payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:56.748921 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.748886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c39b306-58f3-4cab-848b-48f674d404a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:56.749098 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.748935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52t2\" (UniqueName: \"kubernetes.io/projected/7c39b306-58f3-4cab-848b-48f674d404a4-kube-api-access-r52t2\") pod \"payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:56.749553 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.749533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c39b306-58f3-4cab-848b-48f674d404a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:56.759899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.759878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52t2\" (UniqueName: \"kubernetes.io/projected/7c39b306-58f3-4cab-848b-48f674d404a4-kube-api-access-r52t2\") pod \"payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:56.805633 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:56.805600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:57.081286 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:57.081263    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"]
Sep 14 10:54:57.082794 ip-10-0-133-125 kubenswrapper[2563]: W0914 10:54:57.082766    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c39b306_58f3_4cab_848b_48f674d404a4.slice/crio-26e8004e90dc725f68e1968596567a0119ab6992cbf5b8cd36cb2723d62e1295 WatchSource:0}: Error finding container 26e8004e90dc725f68e1968596567a0119ab6992cbf5b8cd36cb2723d62e1295: Status 404 returned error can't find the container with id 26e8004e90dc725f68e1968596567a0119ab6992cbf5b8cd36cb2723d62e1295
Sep 14 10:54:58.055671 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:58.055628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg" event={"ID":"7c39b306-58f3-4cab-848b-48f674d404a4","Type":"ContainerStarted","Data":"99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef"}
Sep 14 10:54:58.056179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:58.055778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:54:58.056179 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:58.055823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg" event={"ID":"7c39b306-58f3-4cab-848b-48f674d404a4","Type":"ContainerStarted","Data":"26e8004e90dc725f68e1968596567a0119ab6992cbf5b8cd36cb2723d62e1295"}
Sep 14 10:54:58.080810 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:54:58.080765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg" podStartSLOduration=2.08075368 podStartE2EDuration="2.08075368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:54:58.079366696 +0000 UTC m=+1786.634554622" watchObservedRunningTime="2026-09-14 10:54:58.08075368 +0000 UTC m=+1786.635941604"
Sep 14 10:55:01.473136 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:01.473095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:55:02.962876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:02.962836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:09.065338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:09.065303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:55:11.153748 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:11.153723    2563 generic.go:356] "Generic (PLEG): container finished" podID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerID="37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108" exitCode=0
Sep 14 10:55:11.154150 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:11.153761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" event={"ID":"1094696e-6623-4bc0-8ac2-22f1683baab1","Type":"ContainerDied","Data":"37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108"}
Sep 14 10:55:11.154150 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:11.154137    2563 scope.go:117] "RemoveContainer" containerID="37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108"
Sep 14 10:55:11.155296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:11.155268    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:55:11.155355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:11.155340    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:55:11.155393 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:11.155373    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 14 10:55:12.160419 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:12.160383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" event={"ID":"1094696e-6623-4bc0-8ac2-22f1683baab1","Type":"ContainerStarted","Data":"6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1"}
Sep 14 10:55:12.229596 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:12.210635    2563 scope.go:117] "RemoveContainer" containerID="bd71a15a3e71a3d64139d36daafe27de3393401798a076b09d6d742d156aec0a"
Sep 14 10:55:12.782612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:12.782552    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:55:12.782847 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:12.782821    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 14 10:55:12.782899 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:12.782880    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 14 10:55:13.782924 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:13.782883    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 14 10:55:13.783386 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:13.782952    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 14 10:55:14.782478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:14.782449    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 14 10:55:14.782665 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:14.782499    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 14 10:55:15.783085 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:15.783042    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 14 10:55:15.783462 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:15.783104    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 14 10:55:16.782583 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:16.782531    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 14 10:55:16.782738 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:16.782625    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 14 10:55:17.787083 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:17.787049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:55:17.787515 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:17.787300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:55:17.788138 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:17.788120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:55:18.435267 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:18.435227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:19.173256 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:19.173220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:19.853157 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:19.853127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:20.717771 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:20.717741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:21.551415 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:21.551383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:22.055317 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:22.055281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:22.402990 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:22.402959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:23.328955 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:23.328908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:24.068770 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:24.068739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:24.982037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:24.982007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:25.837706 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:25.837649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:26.678771 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:26.678739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:27.847831 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:27.847794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:34.084611 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.084563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:34.202697 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.202664    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"]
Sep 14 10:55:34.202883 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.202862    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg" podUID="7c39b306-58f3-4cab-848b-48f674d404a4" containerName="payload-pre-processing" containerID="cri-o://99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef" gracePeriod=30
Sep 14 10:55:34.270386 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.270350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:34.408037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.408009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:34.447434 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.447414    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:55:34.514438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.514406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c39b306-58f3-4cab-848b-48f674d404a4-plugins-config-volume\") pod \"7c39b306-58f3-4cab-848b-48f674d404a4\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") "
Sep 14 10:55:34.514624 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.514531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r52t2\" (UniqueName: \"kubernetes.io/projected/7c39b306-58f3-4cab-848b-48f674d404a4-kube-api-access-r52t2\") pod \"7c39b306-58f3-4cab-848b-48f674d404a4\" (UID: \"7c39b306-58f3-4cab-848b-48f674d404a4\") "
Sep 14 10:55:34.514952 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.514919    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c39b306-58f3-4cab-848b-48f674d404a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c39b306-58f3-4cab-848b-48f674d404a4" (UID: "7c39b306-58f3-4cab-848b-48f674d404a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:55:34.516842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.516818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c39b306-58f3-4cab-848b-48f674d404a4-kube-api-access-r52t2" (OuterVolumeSpecName: "kube-api-access-r52t2") pod "7c39b306-58f3-4cab-848b-48f674d404a4" (UID: "7c39b306-58f3-4cab-848b-48f674d404a4"). InnerVolumeSpecName "kube-api-access-r52t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:55:34.556332 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.556298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:34.615822 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.615761    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r52t2\" (UniqueName: \"kubernetes.io/projected/7c39b306-58f3-4cab-848b-48f674d404a4-kube-api-access-r52t2\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:34.615822 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.615785    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c39b306-58f3-4cab-848b-48f674d404a4-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:34.665483 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:34.665444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:35.035466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.035431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:35.196209 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.196184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:35.277866 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.277836    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c39b306-58f3-4cab-848b-48f674d404a4" containerID="99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef" exitCode=0
Sep 14 10:55:35.278037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.277885    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"
Sep 14 10:55:35.278037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.277904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg" event={"ID":"7c39b306-58f3-4cab-848b-48f674d404a4","Type":"ContainerDied","Data":"99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef"}
Sep 14 10:55:35.278037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.277948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg" event={"ID":"7c39b306-58f3-4cab-848b-48f674d404a4","Type":"ContainerDied","Data":"26e8004e90dc725f68e1968596567a0119ab6992cbf5b8cd36cb2723d62e1295"}
Sep 14 10:55:35.278037 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.277972    2563 scope.go:117] "RemoveContainer" containerID="99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef"
Sep 14 10:55:35.286950 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.286926    2563 scope.go:117] "RemoveContainer" containerID="99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef"
Sep 14 10:55:35.287221 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:55:35.287203    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef\": container with ID starting with 99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef not found: ID does not exist" containerID="99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef"
Sep 14 10:55:35.287272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.287229    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef"} err="failed to get container status \"99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef\": rpc error: code = NotFound desc = could not find container \"99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef\": container with ID starting with 99b81395289b9d76b49b5933716ebe36c53ebc269c0eb605b0e6d1deaab033ef not found: ID does not exist"
Sep 14 10:55:35.319488 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.319457    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"]
Sep 14 10:55:35.326093 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.326067    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7599d7-7974d5f678-x5rcg"]
Sep 14 10:55:35.919153 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:35.919110    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c39b306-58f3-4cab-848b-48f674d404a4" path="/var/lib/kubelet/pods/7c39b306-58f3-4cab-848b-48f674d404a4/volumes"
Sep 14 10:55:36.220821 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:36.220737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:37.134372 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:37.134345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:37.926635 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:37.926603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:38.754021 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:38.753990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:38.892976 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:38.892947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:55:41.279935 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:41.279905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:41.381072 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:41.381038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:41.531006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:41.530925    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"]
Sep 14 10:55:41.531167 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:41.531143    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy" containerID="cri-o://6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1" gracePeriod=30
Sep 14 10:55:41.557049 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:41.557021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:45.278488 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:45.278449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:46.180264 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.180228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:46.780542 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.780521    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:55:46.894481 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.894447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:46.918966 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.918938    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-data\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.918995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-podinfo\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-certs\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919067 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1094696e-6623-4bc0-8ac2-22f1683baab1-istiod-ca-cert\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-credential-socket\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8r8f\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-kube-api-access-p8r8f\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-token\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919188 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-envoy\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-socket\") pod \"1094696e-6623-4bc0-8ac2-22f1683baab1\" (UID: \"1094696e-6623-4bc0-8ac2-22f1683baab1\") "
Sep 14 10:55:46.919363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919210    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-data" (OuterVolumeSpecName: "istio-data") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:46.919363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:46.919363 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919346    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:46.919565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919524    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:46.919565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919544    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:46.919565 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919557    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:46.919770 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919614    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:46.919824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.919800    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1094696e-6623-4bc0-8ac2-22f1683baab1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:55:46.921411 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.921377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:55:46.921411 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.921399    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-token" (OuterVolumeSpecName: "istio-token") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:55:46.921524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.921405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:55:46.921524 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:46.921429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-kube-api-access-p8r8f" (OuterVolumeSpecName: "kube-api-access-p8r8f") pod "1094696e-6623-4bc0-8ac2-22f1683baab1" (UID: "1094696e-6623-4bc0-8ac2-22f1683baab1"). InnerVolumeSpecName "kube-api-access-p8r8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:55:47.020811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.020779    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8r8f\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-kube-api-access-p8r8f\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:47.020811 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.020811    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:47.021006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.020825    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:47.021006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.020836    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1094696e-6623-4bc0-8ac2-22f1683baab1-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:47.021006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.020847    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1094696e-6623-4bc0-8ac2-22f1683baab1-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:47.021006 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.020859    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1094696e-6623-4bc0-8ac2-22f1683baab1-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:55:47.347296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.347212    2563 generic.go:356] "Generic (PLEG): container finished" podID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerID="6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1" exitCode=0
Sep 14 10:55:47.347296 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.347262    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"
Sep 14 10:55:47.347472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.347297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" event={"ID":"1094696e-6623-4bc0-8ac2-22f1683baab1","Type":"ContainerDied","Data":"6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1"}
Sep 14 10:55:47.347472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.347356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4" event={"ID":"1094696e-6623-4bc0-8ac2-22f1683baab1","Type":"ContainerDied","Data":"87f83eaec89431802595a5a67526f4b74029ffde7c570884a321425c401c8e1a"}
Sep 14 10:55:47.347472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.347384    2563 scope.go:117] "RemoveContainer" containerID="6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1"
Sep 14 10:55:47.357407 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.357389    2563 scope.go:117] "RemoveContainer" containerID="37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108"
Sep 14 10:55:47.365464 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.365448    2563 scope.go:117] "RemoveContainer" containerID="6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1"
Sep 14 10:55:47.365758 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:55:47.365737    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1\": container with ID starting with 6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1 not found: ID does not exist" containerID="6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1"
Sep 14 10:55:47.365843 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.365764    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1"} err="failed to get container status \"6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1\": rpc error: code = NotFound desc = could not find container \"6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1\": container with ID starting with 6a72aa79606789360d92090f32be7522ab780cdace41796024b017bdbd90a9a1 not found: ID does not exist"
Sep 14 10:55:47.365843 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.365786    2563 scope.go:117] "RemoveContainer" containerID="37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108"
Sep 14 10:55:47.366116 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:55:47.366088    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108\": container with ID starting with 37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108 not found: ID does not exist" containerID="37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108"
Sep 14 10:55:47.366221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.366123    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108"} err="failed to get container status \"37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108\": rpc error: code = NotFound desc = could not find container \"37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108\": container with ID starting with 37290e21d3859dcdd0d11259e121fb3a10a11fd5fb0750f9b6c1588ef4b86108 not found: ID does not exist"
Sep 14 10:55:47.368492 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.368468    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"]
Sep 14 10:55:47.372181 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.372159    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7599d7-openshift-default-858c8888c4-8pdl4"]
Sep 14 10:55:47.918990 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:47.918956    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" path="/var/lib/kubelet/pods/1094696e-6623-4bc0-8ac2-22f1683baab1/volumes"
Sep 14 10:55:48.142426 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:48.142400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:48.310832 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:48.310767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:49.069885 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:49.069857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:55:58.484007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:55:58.483972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:56:00.571629 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.571594    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"]
Sep 14 10:56:00.572026 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.571810    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r" podUID="3c757dfb-a14d-4903-a48a-db2674ff7698" containerName="maas-api" containerID="cri-o://31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41" gracePeriod=30
Sep 14 10:56:00.618466 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.618439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:00.715612 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.715564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:00.796593 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.796542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:00.824228 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.824171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:56:00.832656 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.832631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-maas-api-tls\") pod \"3c757dfb-a14d-4903-a48a-db2674ff7698\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") "
Sep 14 10:56:00.832783 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.832719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-metrics-certs\") pod \"3c757dfb-a14d-4903-a48a-db2674ff7698\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") "
Sep 14 10:56:00.832783 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.832755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2vxw\" (UniqueName: \"kubernetes.io/projected/3c757dfb-a14d-4903-a48a-db2674ff7698-kube-api-access-n2vxw\") pod \"3c757dfb-a14d-4903-a48a-db2674ff7698\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") "
Sep 14 10:56:00.832865 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.832788    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c757dfb-a14d-4903-a48a-db2674ff7698-openshift-service-ca\") pod \"3c757dfb-a14d-4903-a48a-db2674ff7698\" (UID: \"3c757dfb-a14d-4903-a48a-db2674ff7698\") "
Sep 14 10:56:00.833248 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.833214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c757dfb-a14d-4903-a48a-db2674ff7698-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c757dfb-a14d-4903-a48a-db2674ff7698" (UID: "3c757dfb-a14d-4903-a48a-db2674ff7698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:56:00.834849 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.834830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3c757dfb-a14d-4903-a48a-db2674ff7698" (UID: "3c757dfb-a14d-4903-a48a-db2674ff7698"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:56:00.834946 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.834850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c757dfb-a14d-4903-a48a-db2674ff7698" (UID: "3c757dfb-a14d-4903-a48a-db2674ff7698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:56:00.835007 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.834963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c757dfb-a14d-4903-a48a-db2674ff7698-kube-api-access-n2vxw" (OuterVolumeSpecName: "kube-api-access-n2vxw") pod "3c757dfb-a14d-4903-a48a-db2674ff7698" (UID: "3c757dfb-a14d-4903-a48a-db2674ff7698"). InnerVolumeSpecName "kube-api-access-n2vxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:56:00.883338 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.883307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:00.934015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.933983    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.934015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.934012    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vxw\" (UniqueName: \"kubernetes.io/projected/3c757dfb-a14d-4903-a48a-db2674ff7698-kube-api-access-n2vxw\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.934015 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.934021    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c757dfb-a14d-4903-a48a-db2674ff7698-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:00.934255 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:00.934030    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c757dfb-a14d-4903-a48a-db2674ff7698-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:01.274507 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.274474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:01.388662 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.388620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:01.421576 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.421544    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c757dfb-a14d-4903-a48a-db2674ff7698" containerID="31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41" exitCode=0
Sep 14 10:56:01.421735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.421617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"
Sep 14 10:56:01.421735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.421629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r" event={"ID":"3c757dfb-a14d-4903-a48a-db2674ff7698","Type":"ContainerDied","Data":"31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41"}
Sep 14 10:56:01.421735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.421686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r" event={"ID":"3c757dfb-a14d-4903-a48a-db2674ff7698","Type":"ContainerDied","Data":"e3d12229240655141401ce0608fec88ab4899dceb165732b54744ff9e0f46674"}
Sep 14 10:56:01.421735 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.421714    2563 scope.go:117] "RemoveContainer" containerID="31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41"
Sep 14 10:56:01.431208 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.431192    2563 scope.go:117] "RemoveContainer" containerID="31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41"
Sep 14 10:56:01.431461 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:56:01.431441    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41\": container with ID starting with 31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41 not found: ID does not exist" containerID="31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41"
Sep 14 10:56:01.431513 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.431468    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41"} err="failed to get container status \"31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41\": rpc error: code = NotFound desc = could not find container \"31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41\": container with ID starting with 31ca11c72dccda8bd20d0b5e3a46444a22c77e232fe3b56edea5a6dcec9ddb41 not found: ID does not exist"
Sep 14 10:56:01.447287 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.447264    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"]
Sep 14 10:56:01.450137 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.450115    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-689fac-c7956f796-j5g4r"]
Sep 14 10:56:01.919272 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:01.919230    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c757dfb-a14d-4903-a48a-db2674ff7698" path="/var/lib/kubelet/pods/3c757dfb-a14d-4903-a48a-db2674ff7698/volumes"
Sep 14 10:56:02.357053 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:02.357024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:03.202364 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:03.202328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:03.967955 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:03.967921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:06.247381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:06.247347    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"]
Sep 14 10:56:06.340221 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:06.340194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:10.477415 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:10.477378    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:56:21.266328 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.266287    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w" podUID="c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" containerName="main" containerID="cri-o://1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c" gracePeriod=45
Sep 14 10:56:21.509690 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.509666    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:56:21.518412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.518362    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" containerID="1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c" exitCode=0
Sep 14 10:56:21.518412 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.518397    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"
Sep 14 10:56:21.518561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.518452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w" event={"ID":"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e","Type":"ContainerDied","Data":"1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c"}
Sep 14 10:56:21.518561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.518493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w" event={"ID":"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e","Type":"ContainerDied","Data":"9c21242ac26b81c17d7da16533c3b1992d683a701a81dc34143e432b7e7ed8a1"}
Sep 14 10:56:21.518561 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.518511    2563 scope.go:117] "RemoveContainer" containerID="1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c"
Sep 14 10:56:21.526798 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.526779    2563 scope.go:117] "RemoveContainer" containerID="1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c"
Sep 14 10:56:21.527227 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:56:21.527200    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c\": container with ID starting with 1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c not found: ID does not exist" containerID="1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c"
Sep 14 10:56:21.527373 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.527240    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c"} err="failed to get container status \"1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c\": rpc error: code = NotFound desc = could not find container \"1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c\": container with ID starting with 1a7597bb1acff47b9d684b52bc09aecdc00af18ae0a909e0db4c647ce8755c2c not found: ID does not exist"
Sep 14 10:56:21.602416 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602388    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-home\") pod \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") "
Sep 14 10:56:21.602626 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602430    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tls-certs\") pod \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") "
Sep 14 10:56:21.602626 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602502    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwc8r\" (UniqueName: \"kubernetes.io/projected/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-kube-api-access-kwc8r\") pod \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") "
Sep 14 10:56:21.602626 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602546    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-model-cache\") pod \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") "
Sep 14 10:56:21.602626 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-dshm\") pod \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") "
Sep 14 10:56:21.602840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tmp-dir\") pod \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\" (UID: \"c8a35ef6-10f7-4185-9f85-bbdd381d2c2e\") "
Sep 14 10:56:21.602840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-home" (OuterVolumeSpecName: "home") pod "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" (UID: "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:56:21.602930 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" (UID: "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:56:21.602974 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.602955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" (UID: "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:56:21.603051 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.603034    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:21.603100 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.603057    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:21.603100 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.603066    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:21.604774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.604746    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-dshm" (OuterVolumeSpecName: "dshm") pod "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" (UID: "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:56:21.604875 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.604790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" (UID: "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:56:21.604875 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.604830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-kube-api-access-kwc8r" (OuterVolumeSpecName: "kube-api-access-kwc8r") pod "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" (UID: "c8a35ef6-10f7-4185-9f85-bbdd381d2c2e"). InnerVolumeSpecName "kube-api-access-kwc8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:56:21.704514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.704488    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwc8r\" (UniqueName: \"kubernetes.io/projected/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-kube-api-access-kwc8r\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:21.704514 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.704512    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:21.704681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.704522    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:56:21.842059 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.842025    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"]
Sep 14 10:56:21.845130 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.845105    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-689fac/ipp-route-689fac-kserve-8f79f5c9f-b7s7w"]
Sep 14 10:56:21.918389 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:21.918361    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" path="/var/lib/kubelet/pods/c8a35ef6-10f7-4185-9f85-bbdd381d2c2e/volumes"
Sep 14 10:56:42.730801 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:42.730721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:43.617623 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:43.617588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:49.202165 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:49.202135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:49.301377 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:49.301323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:49.463932 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:49.463860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:50.683854 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:50.683819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:51.350968 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:51.350933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:52.089889 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:52.089852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:56:52.898959 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:56:52.898921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:00.393638 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.393604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:00.394010 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.393920    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"]
Sep 14 10:57:00.394168 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.394127    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v" podUID="02fb1c4b-b678-40a1-8202-64ee6efc8f6a" containerName="payload-pre-processing" containerID="cri-o://10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54" gracePeriod=30
Sep 14 10:57:00.475381 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.475350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:00.567388 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.567356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:00.635920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.635891    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:57:00.643139 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.643106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:00.708991 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.708915    2563 generic.go:356] "Generic (PLEG): container finished" podID="02fb1c4b-b678-40a1-8202-64ee6efc8f6a" containerID="10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54" exitCode=0
Sep 14 10:57:00.709128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.709001    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"
Sep 14 10:57:00.709128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.709038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v" event={"ID":"02fb1c4b-b678-40a1-8202-64ee6efc8f6a","Type":"ContainerDied","Data":"10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54"}
Sep 14 10:57:00.709128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.709093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v" event={"ID":"02fb1c4b-b678-40a1-8202-64ee6efc8f6a","Type":"ContainerDied","Data":"b488064e5f59d533a31e30e93d3866fc958660f86346fcc7d0576242ca44073d"}
Sep 14 10:57:00.709128 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.709121    2563 scope.go:117] "RemoveContainer" containerID="10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54"
Sep 14 10:57:00.717433 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.717416    2563 scope.go:117] "RemoveContainer" containerID="10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54"
Sep 14 10:57:00.717713 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:57:00.717693    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54\": container with ID starting with 10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54 not found: ID does not exist" containerID="10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54"
Sep 14 10:57:00.717789 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.717722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54"} err="failed to get container status \"10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54\": rpc error: code = NotFound desc = could not find container \"10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54\": container with ID starting with 10ae34b3a7fdbe8cf19b40666fee08ede6c8d78f3b23a68214bb85779e49dc54 not found: ID does not exist"
Sep 14 10:57:00.816681 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.816654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-plugins-config-volume\") pod \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") "
Sep 14 10:57:00.816830 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.816694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnn9q\" (UniqueName: \"kubernetes.io/projected/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-kube-api-access-cnn9q\") pod \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\" (UID: \"02fb1c4b-b678-40a1-8202-64ee6efc8f6a\") "
Sep 14 10:57:00.817066 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.817038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02fb1c4b-b678-40a1-8202-64ee6efc8f6a" (UID: "02fb1c4b-b678-40a1-8202-64ee6efc8f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:57:00.818740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.818722    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-kube-api-access-cnn9q" (OuterVolumeSpecName: "kube-api-access-cnn9q") pod "02fb1c4b-b678-40a1-8202-64ee6efc8f6a" (UID: "02fb1c4b-b678-40a1-8202-64ee6efc8f6a"). InnerVolumeSpecName "kube-api-access-cnn9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:00.917740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.917720    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:00.917740 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.917741    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnn9q\" (UniqueName: \"kubernetes.io/projected/02fb1c4b-b678-40a1-8202-64ee6efc8f6a-kube-api-access-cnn9q\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:00.942968 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:00.942943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:01.029107 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:01.029071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"]
Sep 14 10:57:01.039474 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:01.039438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:01.044817 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:01.044796    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7406d4-6d959666c8-9826v"]
Sep 14 10:57:01.685422 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:01.685384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:01.920081 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:01.920044    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02fb1c4b-b678-40a1-8202-64ee6efc8f6a" path="/var/lib/kubelet/pods/02fb1c4b-b678-40a1-8202-64ee6efc8f6a/volumes"
Sep 14 10:57:02.524293 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:02.524221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:03.762267 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:03.762234    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:57:06.808877 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:06.808845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:06.911747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:06.911715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:07.060684 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:07.060596    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"]
Sep 14 10:57:07.060842 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:07.060788    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy" containerID="cri-o://46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9" gracePeriod=30
Sep 14 10:57:07.086890 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:07.086863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:07.585722 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:07.585682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:08.488778 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:08.488739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:09.308001 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:09.307968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:09.985024 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:09.984984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:12.299840 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.299819    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:57:12.419094 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-data\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsl5g\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-kube-api-access-gsl5g\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419129    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-token\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-certs\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-socket\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419258 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-credential-socket\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istiod-ca-cert\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-podinfo\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-envoy\") pod \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\" (UID: \"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4\") "
Sep 14 10:57:12.419516 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-data" (OuterVolumeSpecName: "istio-data") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:12.419744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:12.419744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:12.419744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:57:12.419744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419667    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.419744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419684    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.419744 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419696    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.420020 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.419847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:12.421512 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.421482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-kube-api-access-gsl5g" (OuterVolumeSpecName: "kube-api-access-gsl5g") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "kube-api-access-gsl5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:12.421658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.421516    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:57:12.421658 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.421539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-token" (OuterVolumeSpecName: "istio-token") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:12.421747 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.421656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" (UID: "eaeb9c1e-5a22-48c0-94bb-4c0525559bc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:12.521112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.521069    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.521112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.521108    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.521112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.521118    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.521112 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.521126    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.521379 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.521136    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsl5g\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-kube-api-access-gsl5g\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.521379 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.521143    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:12.768020 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.767935    2563 generic.go:356] "Generic (PLEG): container finished" podID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerID="46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9" exitCode=0
Sep 14 10:57:12.768020 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.767990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" event={"ID":"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4","Type":"ContainerDied","Data":"46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9"}
Sep 14 10:57:12.768257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.767994    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"
Sep 14 10:57:12.768257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.768069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh" event={"ID":"eaeb9c1e-5a22-48c0-94bb-4c0525559bc4","Type":"ContainerDied","Data":"c60c75f7057ded0cb7ecfd92937b984a58293587431d2c4197fa5e5ada7f1c4e"}
Sep 14 10:57:12.768257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.768098    2563 scope.go:117] "RemoveContainer" containerID="46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9"
Sep 14 10:57:12.776931 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.776914    2563 scope.go:117] "RemoveContainer" containerID="aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096"
Sep 14 10:57:12.784790 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.784775    2563 scope.go:117] "RemoveContainer" containerID="46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9"
Sep 14 10:57:12.785051 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:57:12.785032    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9\": container with ID starting with 46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9 not found: ID does not exist" containerID="46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9"
Sep 14 10:57:12.785125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.785063    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9"} err="failed to get container status \"46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9\": rpc error: code = NotFound desc = could not find container \"46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9\": container with ID starting with 46f47798708a61ac54c849427a6acd0ca21cbef66fd2a2e3d69a9d8eb055b6b9 not found: ID does not exist"
Sep 14 10:57:12.785125 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.785088    2563 scope.go:117] "RemoveContainer" containerID="aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096"
Sep 14 10:57:12.785317 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:57:12.785302    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096\": container with ID starting with aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096 not found: ID does not exist" containerID="aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096"
Sep 14 10:57:12.785355 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.785323    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096"} err="failed to get container status \"aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096\": rpc error: code = NotFound desc = could not find container \"aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096\": container with ID starting with aad0bd65d2d5f20aae31d4fb3d0fed6d984756fd8025f6ab9ba647b15cdda096 not found: ID does not exist"
Sep 14 10:57:12.789749 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.789730    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"]
Sep 14 10:57:12.793271 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:12.793253    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7406d4-openshift-default-79fbf5898-j2jxh"]
Sep 14 10:57:13.917540 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:13.917507    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" path="/var/lib/kubelet/pods/eaeb9c1e-5a22-48c0-94bb-4c0525559bc4/volumes"
Sep 14 10:57:17.967670 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:17.967638    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"]
Sep 14 10:57:17.968126 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:17.967898    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx" podUID="b8161d19-36ea-4edc-a29c-eaa7890d5238" containerName="maas-api" containerID="cri-o://cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b" gracePeriod=30
Sep 14 10:57:17.996779 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:17.996721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:18.068069 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.068025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:18.171472 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.171441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:18.212774 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.212751    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:57:18.226717 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.226663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:18.272726 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.272693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44znj\" (UniqueName: \"kubernetes.io/projected/b8161d19-36ea-4edc-a29c-eaa7890d5238-kube-api-access-44znj\") pod \"b8161d19-36ea-4edc-a29c-eaa7890d5238\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") "
Sep 14 10:57:18.272903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.272736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls\") pod \"b8161d19-36ea-4edc-a29c-eaa7890d5238\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") "
Sep 14 10:57:18.272903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.272805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-metrics-certs\") pod \"b8161d19-36ea-4edc-a29c-eaa7890d5238\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") "
Sep 14 10:57:18.272903 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.272827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8161d19-36ea-4edc-a29c-eaa7890d5238-openshift-service-ca\") pod \"b8161d19-36ea-4edc-a29c-eaa7890d5238\" (UID: \"b8161d19-36ea-4edc-a29c-eaa7890d5238\") "
Sep 14 10:57:18.273218 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.273194    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8161d19-36ea-4edc-a29c-eaa7890d5238-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8161d19-36ea-4edc-a29c-eaa7890d5238" (UID: "b8161d19-36ea-4edc-a29c-eaa7890d5238"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:57:18.274902 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.274870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8161d19-36ea-4edc-a29c-eaa7890d5238" (UID: "b8161d19-36ea-4edc-a29c-eaa7890d5238"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:57:18.274902 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.274873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8161d19-36ea-4edc-a29c-eaa7890d5238-kube-api-access-44znj" (OuterVolumeSpecName: "kube-api-access-44znj") pod "b8161d19-36ea-4edc-a29c-eaa7890d5238" (UID: "b8161d19-36ea-4edc-a29c-eaa7890d5238"). InnerVolumeSpecName "kube-api-access-44znj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:18.275039 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.274921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b8161d19-36ea-4edc-a29c-eaa7890d5238" (UID: "b8161d19-36ea-4edc-a29c-eaa7890d5238"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 10:57:18.374088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.374049    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44znj\" (UniqueName: \"kubernetes.io/projected/b8161d19-36ea-4edc-a29c-eaa7890d5238-kube-api-access-44znj\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:18.374088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.374083    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:18.374088 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.374093    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8161d19-36ea-4edc-a29c-eaa7890d5238-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:18.374325 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.374100    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8161d19-36ea-4edc-a29c-eaa7890d5238-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:18.614198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.614120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:18.707378 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.707345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:18.797253 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.797222    2563 generic.go:356] "Generic (PLEG): container finished" podID="b8161d19-36ea-4edc-a29c-eaa7890d5238" containerID="cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b" exitCode=0
Sep 14 10:57:18.797413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.797267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx" event={"ID":"b8161d19-36ea-4edc-a29c-eaa7890d5238","Type":"ContainerDied","Data":"cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b"}
Sep 14 10:57:18.797413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.797347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx" event={"ID":"b8161d19-36ea-4edc-a29c-eaa7890d5238","Type":"ContainerDied","Data":"f1aa94304992038215f80a72bd7a8b2427c49bcb85aa32a53f371a3719ac08a9"}
Sep 14 10:57:18.797413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.797275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"
Sep 14 10:57:18.797413 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.797370    2563 scope.go:117] "RemoveContainer" containerID="cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b"
Sep 14 10:57:18.809895 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.809875    2563 scope.go:117] "RemoveContainer" containerID="cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b"
Sep 14 10:57:18.810142 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:57:18.810125    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b\": container with ID starting with cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b not found: ID does not exist" containerID="cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b"
Sep 14 10:57:18.810197 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.810149    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b"} err="failed to get container status \"cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b\": rpc error: code = NotFound desc = could not find container \"cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b\": container with ID starting with cd213ba4f4e9ac35e070eeaa46e8acd19b235581ccfc9481ed7db1cdd55a0b9b not found: ID does not exist"
Sep 14 10:57:18.820378 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.820357    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"]
Sep 14 10:57:18.824502 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:18.824480    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1cbcd0-86bb49cd55-pqsbx"]
Sep 14 10:57:19.407438 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:19.407406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:19.917043 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:19.917013    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8161d19-36ea-4edc-a29c-eaa7890d5238" path="/var/lib/kubelet/pods/b8161d19-36ea-4edc-a29c-eaa7890d5238/volumes"
Sep 14 10:57:21.169711 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:21.169679    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:57:24.617526 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:24.617486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:24.697243 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:24.697210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:24.906718 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:24.906685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:25.599645 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:25.599613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:26.404365 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:26.404333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:27.082422 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:27.082387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:35.933699 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:35.933669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:35.976312 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:35.976270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"]
Sep 14 10:57:35.976558 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:35.976532    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts" podUID="3b5f82b8-1414-418b-b895-c8c2a2125ba3" containerName="payload-pre-processing" containerID="cri-o://a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93" gracePeriod=30
Sep 14 10:57:36.015974 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.015939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:36.116587 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.116543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:36.195795 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.195724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:36.226257 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.226235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:57:36.314652 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.314623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f75w\" (UniqueName: \"kubernetes.io/projected/3b5f82b8-1414-418b-b895-c8c2a2125ba3-kube-api-access-6f75w\") pod \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") "
Sep 14 10:57:36.314652 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.314657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5f82b8-1414-418b-b895-c8c2a2125ba3-plugins-config-volume\") pod \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\" (UID: \"3b5f82b8-1414-418b-b895-c8c2a2125ba3\") "
Sep 14 10:57:36.315062 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.315030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5f82b8-1414-418b-b895-c8c2a2125ba3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b5f82b8-1414-418b-b895-c8c2a2125ba3" (UID: "3b5f82b8-1414-418b-b895-c8c2a2125ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:57:36.316876 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.316857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5f82b8-1414-418b-b895-c8c2a2125ba3-kube-api-access-6f75w" (OuterVolumeSpecName: "kube-api-access-6f75w") pod "3b5f82b8-1414-418b-b895-c8c2a2125ba3" (UID: "3b5f82b8-1414-418b-b895-c8c2a2125ba3"). InnerVolumeSpecName "kube-api-access-6f75w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:36.415964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.415938    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f75w\" (UniqueName: \"kubernetes.io/projected/3b5f82b8-1414-418b-b895-c8c2a2125ba3-kube-api-access-6f75w\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:36.415964 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.415960    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5f82b8-1414-418b-b895-c8c2a2125ba3-plugins-config-volume\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:36.559647 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.559564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:36.656334 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.656305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:36.881160 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.881127    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b5f82b8-1414-418b-b895-c8c2a2125ba3" containerID="a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93" exitCode=0
Sep 14 10:57:36.881313 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.881180    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"
Sep 14 10:57:36.881313 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.881187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts" event={"ID":"3b5f82b8-1414-418b-b895-c8c2a2125ba3","Type":"ContainerDied","Data":"a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93"}
Sep 14 10:57:36.881313 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.881251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts" event={"ID":"3b5f82b8-1414-418b-b895-c8c2a2125ba3","Type":"ContainerDied","Data":"850f189e9f9c22e237e553d3622553e5a6a34e24c4e28e9bd4b7ed50511638de"}
Sep 14 10:57:36.881313 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.881271    2563 scope.go:117] "RemoveContainer" containerID="a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93"
Sep 14 10:57:36.890429 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.890409    2563 scope.go:117] "RemoveContainer" containerID="a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93"
Sep 14 10:57:36.890715 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:57:36.890696    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93\": container with ID starting with a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93 not found: ID does not exist" containerID="a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93"
Sep 14 10:57:36.890770 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.890723    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93"} err="failed to get container status \"a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93\": rpc error: code = NotFound desc = could not find container \"a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93\": container with ID starting with a7a000454c4e7b3a1512d8eaa4de576edc9223adc2cd583c873f782dab377b93 not found: ID does not exist"
Sep 14 10:57:36.901421 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.901400    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"]
Sep 14 10:57:36.907560 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:36.907539    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7bdcda-644fb84d54-wnvts"]
Sep 14 10:57:37.917630 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:37.917597    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b5f82b8-1414-418b-b895-c8c2a2125ba3" path="/var/lib/kubelet/pods/3b5f82b8-1414-418b-b895-c8c2a2125ba3/volumes"
Sep 14 10:57:38.761831 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:38.761795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:57:42.803672 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:42.803645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:42.916748 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:42.916715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:43.104176 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:43.104091    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"]
Sep 14 10:57:43.104334 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:43.104312    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy" containerID="cri-o://653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5" gracePeriod=30
Sep 14 10:57:43.113678 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:43.113653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:57:47.869663 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:47.869628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:57:48.359198 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.359171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:57:48.518504 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-data\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518504 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518463    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-certs\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518504 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-socket\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518504 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518508    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nh2d\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-kube-api-access-9nh2d\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-token\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-credential-socket\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istiod-ca-cert\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518678    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-podinfo\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-envoy\") pod \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\" (UID: \"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0\") "
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518803    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:48.518852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518824    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:48.519251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.518985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:48.519251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.519063    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.519251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.519080    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.519251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.519094    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.519251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.519105    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.519251 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.519112    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 10:57:48.520920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.520897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 10:57:48.520920 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.520912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:48.521071 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.520970    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 14 10:57:48.521071 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.520976    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-kube-api-access-9nh2d" (OuterVolumeSpecName: "kube-api-access-9nh2d") pod "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" (UID: "48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0"). InnerVolumeSpecName "kube-api-access-9nh2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 10:57:48.620114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.620059    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nh2d\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-kube-api-access-9nh2d\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.620114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.620108    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.620114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.620117    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.620114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.620125    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.620114 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.620133    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 10:57:48.940421 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.940390    2563 generic.go:356] "Generic (PLEG): container finished" podID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerID="653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5" exitCode=0
Sep 14 10:57:48.940824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.940435    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"
Sep 14 10:57:48.940824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.940466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" event={"ID":"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0","Type":"ContainerDied","Data":"653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5"}
Sep 14 10:57:48.940824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.940513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz" event={"ID":"48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0","Type":"ContainerDied","Data":"bfb45d4a7802b50ce305345b86d861d75619a424a0dc83c75240355ad6764883"}
Sep 14 10:57:48.940824 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.940534    2563 scope.go:117] "RemoveContainer" containerID="653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5"
Sep 14 10:57:48.950999 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.950897    2563 scope.go:117] "RemoveContainer" containerID="3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4"
Sep 14 10:57:48.958768 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.958751    2563 scope.go:117] "RemoveContainer" containerID="653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5"
Sep 14 10:57:48.959012 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:57:48.958993    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5\": container with ID starting with 653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5 not found: ID does not exist" containerID="653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5"
Sep 14 10:57:48.959056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.959021    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5"} err="failed to get container status \"653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5\": rpc error: code = NotFound desc = could not find container \"653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5\": container with ID starting with 653663981ffd0e42b10e48fd3cf49983541af5144901fa13da5b68a05a8ee1e5 not found: ID does not exist"
Sep 14 10:57:48.959056 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.959037    2563 scope.go:117] "RemoveContainer" containerID="3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4"
Sep 14 10:57:48.959298 ip-10-0-133-125 kubenswrapper[2563]: E0914 10:57:48.959278    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4\": container with ID starting with 3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4 not found: ID does not exist" containerID="3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4"
Sep 14 10:57:48.959346 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.959303    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4"} err="failed to get container status \"3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4\": rpc error: code = NotFound desc = could not find container \"3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4\": container with ID starting with 3d918fba83235e83abf5838d1de4a2d27b8544c62661c6eda542be0a1f1ebfa4 not found: ID does not exist"
Sep 14 10:57:48.967094 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.963430    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"]
Sep 14 10:57:48.969019 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:48.968995    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7bdcda-openshift-default-549b4b9bbd-fr9hz"]
Sep 14 10:57:49.918306 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:57:49.918271    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" path="/var/lib/kubelet/pods/48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0/volumes"
Sep 14 10:58:34.058220 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:58:34.058189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:59:08.485468 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:08.485434    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:59:09.555478 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:09.555442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:59:11.099852 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:11.099822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:59:35.362436 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:35.362354    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:59:36.419095 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:36.419065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 10:59:42.695284 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:42.695251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:59:48.968844 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:48.968812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 10:59:56.474966 ip-10-0-133-125 kubenswrapper[2563]: I0914 10:59:56.474932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:00:14.768232 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:00:14.768197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:00:20.773703 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:00:20.773667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:00:25.427337 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:00:25.427303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:00:25.487818 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:00:25.487794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:00:30.385743 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:00:30.385712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:00:30.485934 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:00:30.485909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:00:35.035151 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:00:35.035110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:07.102122 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:01:07.102038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:39.927033 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:01:39.926996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:40.041478 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:01:40.041445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:01:51.369581 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:01:51.369533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:00.065880 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:00.065837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:08.771330 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771293    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"]
Sep 14 11:02:08.771722 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771671    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c757dfb-a14d-4903-a48a-db2674ff7698" containerName="maas-api"
Sep 14 11:02:08.771722 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771685    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c757dfb-a14d-4903-a48a-db2674ff7698" containerName="maas-api"
Sep 14 11:02:08.771722 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771695    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c39b306-58f3-4cab-848b-48f674d404a4" containerName="payload-pre-processing"
Sep 14 11:02:08.771722 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771700    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c39b306-58f3-4cab-848b-48f674d404a4" containerName="payload-pre-processing"
Sep 14 11:02:08.771722 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771710    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8161d19-36ea-4edc-a29c-eaa7890d5238" containerName="maas-api"
Sep 14 11:02:08.771722 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8161d19-36ea-4edc-a29c-eaa7890d5238" containerName="maas-api"
Sep 14 11:02:08.771722 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771727    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771732    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771747    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771752    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771758    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771767    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02fb1c4b-b678-40a1-8202-64ee6efc8f6a" containerName="payload-pre-processing"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771772    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02fb1c4b-b678-40a1-8202-64ee6efc8f6a" containerName="payload-pre-processing"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771777    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" containerName="main"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771782    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" containerName="main"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771789    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5f82b8-1414-418b-b895-c8c2a2125ba3" containerName="payload-pre-processing"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771793    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5f82b8-1414-418b-b895-c8c2a2125ba3" containerName="payload-pre-processing"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771798    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771803    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771810    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771815    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771823    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771827    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771881    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771888    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5f82b8-1414-418b-b895-c8c2a2125ba3" containerName="payload-pre-processing"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771894    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771900    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02fb1c4b-b678-40a1-8202-64ee6efc8f6a" containerName="payload-pre-processing"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771905    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d3eb7e-3abf-4bed-bad1-cf89c7dd77e0" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771911    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c39b306-58f3-4cab-848b-48f674d404a4" containerName="payload-pre-processing"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8161d19-36ea-4edc-a29c-eaa7890d5238" containerName="maas-api"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771920    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a35ef6-10f7-4185-9f85-bbdd381d2c2e" containerName="main"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771926    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaeb9c1e-5a22-48c0-94bb-4c0525559bc4" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771932    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771936    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c757dfb-a14d-4903-a48a-db2674ff7698" containerName="maas-api"
Sep 14 11:02:08.771975 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.771944    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1094696e-6623-4bc0-8ac2-22f1683baab1" containerName="istio-proxy"
Sep 14 11:02:08.774919 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.774902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.777491 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.777469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0a75b08b-kserve-self-signed-certs\""
Sep 14 11:02:08.794166 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.794141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"]
Sep 14 11:02:08.797325 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.797304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tmp-dir\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.797434 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.797343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-home\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.797434 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.797413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-dshm\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.797513 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.797439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz69s\" (UniqueName: \"kubernetes.io/projected/b16e678b-6e4d-4be1-9fcb-7e1aca545712-kube-api-access-zz69s\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.797513 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.797462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-model-cache\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.797513 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.797483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tls-certs\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.824851 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.824829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:08.898346 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-dshm\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898454 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz69s\" (UniqueName: \"kubernetes.io/projected/b16e678b-6e4d-4be1-9fcb-7e1aca545712-kube-api-access-zz69s\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898454 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-model-cache\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898454 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tls-certs\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898454 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tmp-dir\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898667 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-home\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898802 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-model-cache\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898902 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tmp-dir\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.898902 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.898891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-home\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.900762 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.900736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-dshm\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.901002 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.900986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tls-certs\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:08.906395 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:08.906378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz69s\" (UniqueName: \"kubernetes.io/projected/b16e678b-6e4d-4be1-9fcb-7e1aca545712-kube-api-access-zz69s\") pod \"e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") " pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:09.083266 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:09.083196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:09.355734 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:09.355707    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"]
Sep 14 11:02:10.091715 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:10.091684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw" event={"ID":"b16e678b-6e4d-4be1-9fcb-7e1aca545712","Type":"ContainerStarted","Data":"fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39"}
Sep 14 11:02:10.092073 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:10.091724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw" event={"ID":"b16e678b-6e4d-4be1-9fcb-7e1aca545712","Type":"ContainerStarted","Data":"b02f7b60dae73631b50eeef43514b3b7b9b22187a6d942baa56d4ea3e299ef78"}
Sep 14 11:02:10.109882 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:10.109831    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw" podStartSLOduration=2.109817272 podStartE2EDuration="2.109817272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:10.107558087 +0000 UTC m=+2218.662746013" watchObservedRunningTime="2026-09-14 11:02:10.109817272 +0000 UTC m=+2218.665005190"
Sep 14 11:02:11.291431 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:11.291393    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:12.253149 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.253112    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"]
Sep 14 11:02:12.256596 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.256561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.258749 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.258730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0a75b08b-kserve-self-signed-certs\""
Sep 14 11:02:12.268150 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.268129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"]
Sep 14 11:02:12.289484 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.289463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:12.329825 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.329799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tmp-dir\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.330117 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.329830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-model-cache\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.330117 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.329854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tls-certs\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.330117 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.329944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpxs\" (UniqueName: \"kubernetes.io/projected/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-kube-api-access-dbpxs\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.330117 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.329993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-home\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.330117 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.330041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-dshm\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.430950 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.430922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tls-certs\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431071 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.430984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpxs\" (UniqueName: \"kubernetes.io/projected/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-kube-api-access-dbpxs\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431071 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.431047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-home\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431190 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.431083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-dshm\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431190 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.431134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tmp-dir\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431190 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.431160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-model-cache\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431456 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.431431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-home\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431529 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.431451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tmp-dir\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.431529 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.431506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-model-cache\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.433516 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.433495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-dshm\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.433628 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.433602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tls-certs\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.441806 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.441787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpxs\" (UniqueName: \"kubernetes.io/projected/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-kube-api-access-dbpxs\") pod \"e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") " pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.565374 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.565306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:12.834891 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:12.834676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"]
Sep 14 11:02:12.837178 ip-10-0-133-125 kubenswrapper[2563]: W0914 11:02:12.837148    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1af78011_d21b_45b0_8ec1_5dfe7f2d8aef.slice/crio-a6eab6345216326495759f9f9caee23b295fa13911d62c9297dfe1fc3efb9cbd WatchSource:0}: Error finding container a6eab6345216326495759f9f9caee23b295fa13911d62c9297dfe1fc3efb9cbd: Status 404 returned error can't find the container with id a6eab6345216326495759f9f9caee23b295fa13911d62c9297dfe1fc3efb9cbd
Sep 14 11:02:13.112687 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:13.112598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk" event={"ID":"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef","Type":"ContainerStarted","Data":"1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e"}
Sep 14 11:02:13.112687 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:13.112653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk" event={"ID":"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef","Type":"ContainerStarted","Data":"a6eab6345216326495759f9f9caee23b295fa13911d62c9297dfe1fc3efb9cbd"}
Sep 14 11:02:13.129836 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:13.129791    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk" podStartSLOduration=1.129777163 podStartE2EDuration="1.129777163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:13.129495278 +0000 UTC m=+2221.684683204" watchObservedRunningTime="2026-09-14 11:02:13.129777163 +0000 UTC m=+2221.684965091"
Sep 14 11:02:19.083634 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:19.083604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:19.084073 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:19.083655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:19.095822 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:19.095794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:19.149929 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:19.149900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:20.261378 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:20.261338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:22.565942 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:22.565912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:22.566410 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:22.565965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:22.577784 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:22.577761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:23.168800 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:23.168773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:26.577843 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:26.577811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"]
Sep 14 11:02:26.599069 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:26.599044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:28.651245 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:28.651209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"]
Sep 14 11:02:29.165945 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:29.165903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:29.212546 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:29.212510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:02:38.461233 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:38.461146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:41.596814 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.596778    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw" podUID="b16e678b-6e4d-4be1-9fcb-7e1aca545712" containerName="main" containerID="cri-o://fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39" gracePeriod=45
Sep 14 11:02:41.836809 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.836784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:41.973938 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.973915    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-home\") pod \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") "
Sep 14 11:02:41.974070 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.973942    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz69s\" (UniqueName: \"kubernetes.io/projected/b16e678b-6e4d-4be1-9fcb-7e1aca545712-kube-api-access-zz69s\") pod \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") "
Sep 14 11:02:41.974070 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974001    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tls-certs\") pod \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") "
Sep 14 11:02:41.974070 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-model-cache\") pod \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") "
Sep 14 11:02:41.974070 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-dshm\") pod \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") "
Sep 14 11:02:41.974070 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974070    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tmp-dir\") pod \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\" (UID: \"b16e678b-6e4d-4be1-9fcb-7e1aca545712\") "
Sep 14 11:02:41.974333 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974150    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-home" (OuterVolumeSpecName: "home") pod "b16e678b-6e4d-4be1-9fcb-7e1aca545712" (UID: "b16e678b-6e4d-4be1-9fcb-7e1aca545712"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:41.974333 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-model-cache" (OuterVolumeSpecName: "model-cache") pod "b16e678b-6e4d-4be1-9fcb-7e1aca545712" (UID: "b16e678b-6e4d-4be1-9fcb-7e1aca545712"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:41.974333 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974320    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:41.974458 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974336    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:41.974458 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.974342    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b16e678b-6e4d-4be1-9fcb-7e1aca545712" (UID: "b16e678b-6e4d-4be1-9fcb-7e1aca545712"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:41.976273 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.976245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-dshm" (OuterVolumeSpecName: "dshm") pod "b16e678b-6e4d-4be1-9fcb-7e1aca545712" (UID: "b16e678b-6e4d-4be1-9fcb-7e1aca545712"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:41.976373 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.976273    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b16e678b-6e4d-4be1-9fcb-7e1aca545712" (UID: "b16e678b-6e4d-4be1-9fcb-7e1aca545712"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:02:41.976373 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:41.976301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16e678b-6e4d-4be1-9fcb-7e1aca545712-kube-api-access-zz69s" (OuterVolumeSpecName: "kube-api-access-zz69s") pod "b16e678b-6e4d-4be1-9fcb-7e1aca545712" (UID: "b16e678b-6e4d-4be1-9fcb-7e1aca545712"). InnerVolumeSpecName "kube-api-access-zz69s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:02:42.074926 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.074905    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:42.074926 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.074924    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:42.075060 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.075000    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b16e678b-6e4d-4be1-9fcb-7e1aca545712-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:42.075106 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.075093    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz69s\" (UniqueName: \"kubernetes.io/projected/b16e678b-6e4d-4be1-9fcb-7e1aca545712-kube-api-access-zz69s\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:42.249391 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.249318    2563 generic.go:356] "Generic (PLEG): container finished" podID="b16e678b-6e4d-4be1-9fcb-7e1aca545712" containerID="fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39" exitCode=0
Sep 14 11:02:42.249391 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.249361    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"
Sep 14 11:02:42.249550 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.249401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw" event={"ID":"b16e678b-6e4d-4be1-9fcb-7e1aca545712","Type":"ContainerDied","Data":"fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39"}
Sep 14 11:02:42.249550 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.249442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw" event={"ID":"b16e678b-6e4d-4be1-9fcb-7e1aca545712","Type":"ContainerDied","Data":"b02f7b60dae73631b50eeef43514b3b7b9b22187a6d942baa56d4ea3e299ef78"}
Sep 14 11:02:42.249550 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.249460    2563 scope.go:117] "RemoveContainer" containerID="fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39"
Sep 14 11:02:42.257862 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.257847    2563 scope.go:117] "RemoveContainer" containerID="fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39"
Sep 14 11:02:42.258104 ip-10-0-133-125 kubenswrapper[2563]: E0914 11:02:42.258084    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39\": container with ID starting with fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39 not found: ID does not exist" containerID="fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39"
Sep 14 11:02:42.258168 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.258110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39"} err="failed to get container status \"fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39\": rpc error: code = NotFound desc = could not find container \"fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39\": container with ID starting with fe7b06e191d1f40ae6b6e07fce5713a09a62b871126d7d8c17693c0f3b471f39 not found: ID does not exist"
Sep 14 11:02:42.269443 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.269424    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"]
Sep 14 11:02:42.273685 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:42.273665    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0a75b08b-kserve-59d8486cf-xp6nw"]
Sep 14 11:02:43.670405 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.670337    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk" podUID="1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" containerName="main" containerID="cri-o://1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e" gracePeriod=45
Sep 14 11:02:43.908288 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.908259    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:43.917388 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.917364    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b16e678b-6e4d-4be1-9fcb-7e1aca545712" path="/var/lib/kubelet/pods/b16e678b-6e4d-4be1-9fcb-7e1aca545712/volumes"
Sep 14 11:02:43.992053 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.991995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-dshm\") pod \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") "
Sep 14 11:02:43.992152 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tmp-dir\") pod \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") "
Sep 14 11:02:43.992152 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbpxs\" (UniqueName: \"kubernetes.io/projected/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-kube-api-access-dbpxs\") pod \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") "
Sep 14 11:02:43.992152 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tls-certs\") pod \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") "
Sep 14 11:02:43.992292 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-home\") pod \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") "
Sep 14 11:02:43.992359 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-model-cache\") pod \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\" (UID: \"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef\") "
Sep 14 11:02:43.992359 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992335    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" (UID: "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:43.992515 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992494    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-home" (OuterVolumeSpecName: "home") pod "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" (UID: "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:43.992640 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992619    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-model-cache" (OuterVolumeSpecName: "model-cache") pod "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" (UID: "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:43.992704 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992690    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:43.992742 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992710    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:43.992742 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.992723    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:43.994146 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.994128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-kube-api-access-dbpxs" (OuterVolumeSpecName: "kube-api-access-dbpxs") pod "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" (UID: "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef"). InnerVolumeSpecName "kube-api-access-dbpxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:02:43.994626 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.994600    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" (UID: "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:02:43.994626 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:43.994615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-dshm" (OuterVolumeSpecName: "dshm") pod "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" (UID: "1af78011-d21b-45b0-8ec1-5dfe7f2d8aef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 14 11:02:44.094029 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.094007    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:44.094029 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.094027    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbpxs\" (UniqueName: \"kubernetes.io/projected/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-kube-api-access-dbpxs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:44.094144 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.094037    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:02:44.260201 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.260134    2563 generic.go:356] "Generic (PLEG): container finished" podID="1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" containerID="1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e" exitCode=0
Sep 14 11:02:44.260201 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.260171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk" event={"ID":"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef","Type":"ContainerDied","Data":"1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e"}
Sep 14 11:02:44.260201 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.260186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"
Sep 14 11:02:44.260355 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.260202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk" event={"ID":"1af78011-d21b-45b0-8ec1-5dfe7f2d8aef","Type":"ContainerDied","Data":"a6eab6345216326495759f9f9caee23b295fa13911d62c9297dfe1fc3efb9cbd"}
Sep 14 11:02:44.260355 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.260223    2563 scope.go:117] "RemoveContainer" containerID="1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e"
Sep 14 11:02:44.269212 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.269194    2563 scope.go:117] "RemoveContainer" containerID="1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e"
Sep 14 11:02:44.269475 ip-10-0-133-125 kubenswrapper[2563]: E0914 11:02:44.269459    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e\": container with ID starting with 1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e not found: ID does not exist" containerID="1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e"
Sep 14 11:02:44.269519 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.269483    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e"} err="failed to get container status \"1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e\": rpc error: code = NotFound desc = could not find container \"1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e\": container with ID starting with 1f03efc6889f7b7b883213df64f06ede74d21949c6c885ea86cf870012eba98e not found: ID does not exist"
Sep 14 11:02:44.284058 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.284037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"]
Sep 14 11:02:44.288595 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:44.288557    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0a75b08b-kserve-957cbf5dc-5t5sk"]
Sep 14 11:02:45.918051 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:45.918019    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" path="/var/lib/kubelet/pods/1af78011-d21b-45b0-8ec1-5dfe7f2d8aef/volumes"
Sep 14 11:02:46.566718 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.566685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:46.676418 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.676389    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"]
Sep 14 11:02:46.676755 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.676741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" containerName="main"
Sep 14 11:02:46.676822 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.676756    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" containerName="main"
Sep 14 11:02:46.676822 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.676767    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b16e678b-6e4d-4be1-9fcb-7e1aca545712" containerName="main"
Sep 14 11:02:46.676822 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.676773    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b16e678b-6e4d-4be1-9fcb-7e1aca545712" containerName="main"
Sep 14 11:02:46.676822 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.676824    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1af78011-d21b-45b0-8ec1-5dfe7f2d8aef" containerName="main"
Sep 14 11:02:46.676957 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.676834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b16e678b-6e4d-4be1-9fcb-7e1aca545712" containerName="main"
Sep 14 11:02:46.679781 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.679759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.682812 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.682777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bhlx\""
Sep 14 11:02:46.683073 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.682858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 14 11:02:46.683073 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.682785    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 14 11:02:46.683073 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.682780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 14 11:02:46.683073 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.682786    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 14 11:02:46.690899 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.690875    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"]
Sep 14 11:02:46.816207 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.816173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-maas-api-tls\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.816207 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.816207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-metrics-certs\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.816399 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.816232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/cc40abcd-8061-41b7-952e-597063da1130-kube-api-access-2sswm\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.816399 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.816337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40abcd-8061-41b7-952e-597063da1130-openshift-service-ca\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.917258 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.917232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-maas-api-tls\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.917379 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.917264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-metrics-certs\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.917379 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.917302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/cc40abcd-8061-41b7-952e-597063da1130-kube-api-access-2sswm\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.917379 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.917340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40abcd-8061-41b7-952e-597063da1130-openshift-service-ca\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.917965 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.917937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40abcd-8061-41b7-952e-597063da1130-openshift-service-ca\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.919616 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.919591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-maas-api-tls\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.919869 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.919717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-metrics-certs\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.924663 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.924641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/cc40abcd-8061-41b7-952e-597063da1130-kube-api-access-2sswm\") pod \"maas-api-6997c4c9fc-nx57j\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") " pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:46.992699 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:46.992674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:47.266367 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:47.266317    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"]
Sep 14 11:02:47.268777 ip-10-0-133-125 kubenswrapper[2563]: W0914 11:02:47.268751    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc40abcd_8061_41b7_952e_597063da1130.slice/crio-8dc8622a3efbf6201da772736f77e8bed3f5dc3e1b83c28fe87eed2bd8e2e60f WatchSource:0}: Error finding container 8dc8622a3efbf6201da772736f77e8bed3f5dc3e1b83c28fe87eed2bd8e2e60f: Status 404 returned error can't find the container with id 8dc8622a3efbf6201da772736f77e8bed3f5dc3e1b83c28fe87eed2bd8e2e60f
Sep 14 11:02:47.270484 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:47.270466    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 14 11:02:47.275586 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:47.275535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j" event={"ID":"cc40abcd-8061-41b7-952e-597063da1130","Type":"ContainerStarted","Data":"8dc8622a3efbf6201da772736f77e8bed3f5dc3e1b83c28fe87eed2bd8e2e60f"}
Sep 14 11:02:49.286824 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:49.286787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j" event={"ID":"cc40abcd-8061-41b7-952e-597063da1130","Type":"ContainerStarted","Data":"571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92"}
Sep 14 11:02:49.287233 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:49.286845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:49.305820 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:49.305772    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j" podStartSLOduration=2.274764062 podStartE2EDuration="3.305757725s" totalImagesPullingTime="1.030993663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:49.304118631 +0000 UTC m=+2257.859306559" watchObservedRunningTime="2026-09-14 11:02:49.305757725 +0000 UTC m=+2257.860945651"
Sep 14 11:02:53.966657 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:53.966624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:02:55.297018 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.296989    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:02:55.667514 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.667478    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"]
Sep 14 11:02:55.671074 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.671055    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.688146 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.688119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"]
Sep 14 11:02:55.792828 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.792802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5z9w\" (UniqueName: \"kubernetes.io/projected/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-kube-api-access-d5z9w\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.792976 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.792846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-openshift-service-ca\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.792976 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.792942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-maas-api-tls\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.792976 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.792969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-metrics-certs\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.893952 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.893929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-openshift-service-ca\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.894044 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.893974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-maas-api-tls\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.894044 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.893990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-metrics-certs\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.894153 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.894043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5z9w\" (UniqueName: \"kubernetes.io/projected/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-kube-api-access-d5z9w\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.894622 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.894604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-openshift-service-ca\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.896400 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.896380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-metrics-certs\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.896490 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.896418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-maas-api-tls\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.901998 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.901976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5z9w\" (UniqueName: \"kubernetes.io/projected/ed495252-52f2-43a9-bfdc-0a7f2eb0ad11-kube-api-access-d5z9w\") pod \"maas-api-7964b8c79c-ft8lp\" (UID: \"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11\") " pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:55.980846 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:55.980793    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:56.251034 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:56.250955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"]
Sep 14 11:02:56.253722 ip-10-0-133-125 kubenswrapper[2563]: W0914 11:02:56.253691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded495252_52f2_43a9_bfdc_0a7f2eb0ad11.slice/crio-9cedd39511bbdee7f2b414b1569de18574c246dcfc5dfbaa2800cf93ca7aec75 WatchSource:0}: Error finding container 9cedd39511bbdee7f2b414b1569de18574c246dcfc5dfbaa2800cf93ca7aec75: Status 404 returned error can't find the container with id 9cedd39511bbdee7f2b414b1569de18574c246dcfc5dfbaa2800cf93ca7aec75
Sep 14 11:02:56.322010 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:56.321982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp" event={"ID":"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11","Type":"ContainerStarted","Data":"9cedd39511bbdee7f2b414b1569de18574c246dcfc5dfbaa2800cf93ca7aec75"}
Sep 14 11:02:57.328402 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:57.328364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp" event={"ID":"ed495252-52f2-43a9-bfdc-0a7f2eb0ad11","Type":"ContainerStarted","Data":"d62dbcd77e5102f9bb9d620c3cdedff5077e677545970af0f2c30f7ee9e54948"}
Sep 14 11:02:57.328825 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:57.328482    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:02:57.346768 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:02:57.346728    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp" podStartSLOduration=1.8311118629999998 podStartE2EDuration="2.346715377s" totalImagesPullingTime="515.603514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:57.345202698 +0000 UTC m=+2265.900390625" watchObservedRunningTime="2026-09-14 11:02:57.346715377 +0000 UTC m=+2265.901903303"
Sep 14 11:03:01.167044 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:01.167001    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:03:03.337781 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.337751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7964b8c79c-ft8lp"
Sep 14 11:03:03.390484 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.390446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"]
Sep 14 11:03:03.390677 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.390652    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j" podUID="cc40abcd-8061-41b7-952e-597063da1130" containerName="maas-api" containerID="cri-o://571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92" gracePeriod=30
Sep 14 11:03:03.625000 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.624976    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:03:03.653159 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.653131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-metrics-certs\") pod \"cc40abcd-8061-41b7-952e-597063da1130\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") "
Sep 14 11:03:03.653290 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.653171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/cc40abcd-8061-41b7-952e-597063da1130-kube-api-access-2sswm\") pod \"cc40abcd-8061-41b7-952e-597063da1130\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") "
Sep 14 11:03:03.653290 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.653208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40abcd-8061-41b7-952e-597063da1130-openshift-service-ca\") pod \"cc40abcd-8061-41b7-952e-597063da1130\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") "
Sep 14 11:03:03.653404 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.653327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-maas-api-tls\") pod \"cc40abcd-8061-41b7-952e-597063da1130\" (UID: \"cc40abcd-8061-41b7-952e-597063da1130\") "
Sep 14 11:03:03.653533 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.653509    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc40abcd-8061-41b7-952e-597063da1130-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc40abcd-8061-41b7-952e-597063da1130" (UID: "cc40abcd-8061-41b7-952e-597063da1130"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 14 11:03:03.653714 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.653696    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40abcd-8061-41b7-952e-597063da1130-openshift-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:03:03.655256 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.655230    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc40abcd-8061-41b7-952e-597063da1130" (UID: "cc40abcd-8061-41b7-952e-597063da1130"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:03:03.655347 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.655323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc40abcd-8061-41b7-952e-597063da1130-kube-api-access-2sswm" (OuterVolumeSpecName: "kube-api-access-2sswm") pod "cc40abcd-8061-41b7-952e-597063da1130" (UID: "cc40abcd-8061-41b7-952e-597063da1130"). InnerVolumeSpecName "kube-api-access-2sswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 14 11:03:03.655732 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.655712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc40abcd-8061-41b7-952e-597063da1130" (UID: "cc40abcd-8061-41b7-952e-597063da1130"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 14 11:03:03.754067 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.754036    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-metrics-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:03:03.754067 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.754061    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sswm\" (UniqueName: \"kubernetes.io/projected/cc40abcd-8061-41b7-952e-597063da1130-kube-api-access-2sswm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:03:03.754247 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:03.754075    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40abcd-8061-41b7-952e-597063da1130-maas-api-tls\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\""
Sep 14 11:03:04.361260 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.361189    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc40abcd-8061-41b7-952e-597063da1130" containerID="571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92" exitCode=0
Sep 14 11:03:04.361260 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.361225    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"
Sep 14 11:03:04.361260 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.361231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j" event={"ID":"cc40abcd-8061-41b7-952e-597063da1130","Type":"ContainerDied","Data":"571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92"}
Sep 14 11:03:04.361693 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.361287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j" event={"ID":"cc40abcd-8061-41b7-952e-597063da1130","Type":"ContainerDied","Data":"8dc8622a3efbf6201da772736f77e8bed3f5dc3e1b83c28fe87eed2bd8e2e60f"}
Sep 14 11:03:04.361693 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.361312    2563 scope.go:117] "RemoveContainer" containerID="571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92"
Sep 14 11:03:04.369437 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.369420    2563 scope.go:117] "RemoveContainer" containerID="571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92"
Sep 14 11:03:04.369712 ip-10-0-133-125 kubenswrapper[2563]: E0914 11:03:04.369695    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92\": container with ID starting with 571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92 not found: ID does not exist" containerID="571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92"
Sep 14 11:03:04.369777 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.369721    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92"} err="failed to get container status \"571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92\": rpc error: code = NotFound desc = could not find container \"571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92\": container with ID starting with 571b96ad970db91fa5a4380a9a1cb5c0bffb005d687c28fdca73380367938b92 not found: ID does not exist"
Sep 14 11:03:04.380670 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.380640    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"]
Sep 14 11:03:04.386701 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:04.386680    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6997c4c9fc-nx57j"]
Sep 14 11:03:05.917710 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:05.917681    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc40abcd-8061-41b7-952e-597063da1130" path="/var/lib/kubelet/pods/cc40abcd-8061-41b7-952e-597063da1130/volumes"
Sep 14 11:03:09.755466 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:09.755431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:03:17.170427 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:17.170384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-lldrw"]
Sep 14 11:03:20.485709 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:20.485667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b76478db-s82q9_5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84/authorino/0.log"
Sep 14 11:03:26.422856 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:26.422820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757dcccd8b-fsnj2_36389ec5-c492-4589-9823-53de660d9c77/copy-manifests/0.log"
Sep 14 11:03:26.433118 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:26.433078    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757dcccd8b-fsnj2_36389ec5-c492-4589-9823-53de660d9c77/manager/0.log"
Sep 14 11:03:26.922806 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:26.922779    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7964b8c79c-ft8lp_ed495252-52f2-43a9-bfdc-0a7f2eb0ad11/maas-api/0.log"
Sep 14 11:03:27.141802 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:27.141768    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823045-4b5p9_c858bb9d-5247-4e2b-a081-5331a31141d9/cleanup/0.log"
Sep 14 11:03:28.131698 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:28.131659    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b76478db-s82q9_5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84/authorino/0.log"
Sep 14 11:03:28.260728 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:28.260702    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p9vjv_ad840875-a2e3-4c37-846b-6cae5dd1d7c1/manager/0.log"
Sep 14 11:03:28.742876 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:28.742844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-lldrw_3a9f2178-5ca9-4d4f-a1a0-7783d4b493da/limitador/0.log"
Sep 14 11:03:29.543986 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:29.543960    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb779b95-ww4qf_b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33/kube-auth-proxy/0.log"
Sep 14 11:03:30.893113 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:30.893071    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wqjk2_5dc770b8-ce2f-4926-b03b-c914e0672485/main/0.log"
Sep 14 11:03:31.006630 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:31.006594    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dsbl4_f13b256c-7be8-4d28-8fbd-da2b69adeff6/main/0.log"
Sep 14 11:03:36.293435 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:36.293402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:36.391204 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:36.391177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:39.376141 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:39.376096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tbshl_7b35dd04-3950-42a5-9755-03cb4f5059b6/global-pull-secret-syncer/0.log"
Sep 14 11:03:39.429850 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:39.429827    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log"
Sep 14 11:03:39.528886 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:39.528858    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-125.ec2.internal_bf677e6df7b80e07cd34581a27ba68d0/haproxy/0.log"
Sep 14 11:03:44.127389 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:44.127358    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b76478db-s82q9_5f14aba0-3dcf-4a09-ab1a-ca8f7cd4fd84/authorino/0.log"
Sep 14 11:03:44.170641 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:44.170618    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p9vjv_ad840875-a2e3-4c37-846b-6cae5dd1d7c1/manager/0.log"
Sep 14 11:03:44.326614 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:44.326588    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-lldrw_3a9f2178-5ca9-4d4f-a1a0-7783d4b493da/limitador/0.log"
Sep 14 11:03:45.916008 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:45.915978    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8964b79c-02ad-4e1d-a25a-69eefc641450/alertmanager/0.log"
Sep 14 11:03:45.951120 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:45.951097    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8964b79c-02ad-4e1d-a25a-69eefc641450/config-reloader/0.log"
Sep 14 11:03:45.975663 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:45.975632    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8964b79c-02ad-4e1d-a25a-69eefc641450/kube-rbac-proxy-web/0.log"
Sep 14 11:03:46.002595 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.002518    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8964b79c-02ad-4e1d-a25a-69eefc641450/kube-rbac-proxy/0.log"
Sep 14 11:03:46.027227 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.027207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8964b79c-02ad-4e1d-a25a-69eefc641450/kube-rbac-proxy-metric/0.log"
Sep 14 11:03:46.050970 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.050947    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8964b79c-02ad-4e1d-a25a-69eefc641450/prom-label-proxy/0.log"
Sep 14 11:03:46.072360 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.072340    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8964b79c-02ad-4e1d-a25a-69eefc641450/init-config-reloader/0.log"
Sep 14 11:03:46.141725 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.141703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fxb96_d214b683-dc68-4867-905b-91d1ef5b86e4/kube-state-metrics/0.log"
Sep 14 11:03:46.159923 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.159897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fxb96_d214b683-dc68-4867-905b-91d1ef5b86e4/kube-rbac-proxy-main/0.log"
Sep 14 11:03:46.184824 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.184798    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-fxb96_d214b683-dc68-4867-905b-91d1ef5b86e4/kube-rbac-proxy-self/0.log"
Sep 14 11:03:46.456281 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.456259    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2tft_90927451-7697-40f0-8240-f20e21ad38a3/node-exporter/0.log"
Sep 14 11:03:46.490897 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.490867    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2tft_90927451-7697-40f0-8240-f20e21ad38a3/kube-rbac-proxy/0.log"
Sep 14 11:03:46.515520 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.515498    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2tft_90927451-7697-40f0-8240-f20e21ad38a3/init-textfile/0.log"
Sep 14 11:03:46.564156 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.564100    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-plvcb_e4f99e74-1d8a-4cbf-a252-351efc1a4d01/kube-rbac-proxy-main/0.log"
Sep 14 11:03:46.587652 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.587634    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-plvcb_e4f99e74-1d8a-4cbf-a252-351efc1a4d01/kube-rbac-proxy-self/0.log"
Sep 14 11:03:46.616895 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.616877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-plvcb_e4f99e74-1d8a-4cbf-a252-351efc1a4d01/openshift-state-metrics/0.log"
Sep 14 11:03:46.658690 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.658675    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aac552ec-722c-47a4-a5c7-14b5f986c32e/prometheus/0.log"
Sep 14 11:03:46.686886 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.686866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aac552ec-722c-47a4-a5c7-14b5f986c32e/config-reloader/0.log"
Sep 14 11:03:46.729706 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.729688    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aac552ec-722c-47a4-a5c7-14b5f986c32e/thanos-sidecar/0.log"
Sep 14 11:03:46.759990 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.759966    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aac552ec-722c-47a4-a5c7-14b5f986c32e/kube-rbac-proxy-web/0.log"
Sep 14 11:03:46.790388 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.790369    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aac552ec-722c-47a4-a5c7-14b5f986c32e/kube-rbac-proxy/0.log"
Sep 14 11:03:46.819750 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.819730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aac552ec-722c-47a4-a5c7-14b5f986c32e/kube-rbac-proxy-thanos/0.log"
Sep 14 11:03:46.845746 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.845727    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aac552ec-722c-47a4-a5c7-14b5f986c32e/init-config-reloader/0.log"
Sep 14 11:03:46.875864 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.875838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:46.976175 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.976152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvl4z_9b6fabe4-b972-42d0-b1c9-306df81cbb62/konnectivity-agent/0.log" limit=1024
Sep 14 11:03:46.993638 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:46.993622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c676b8b94-c2zxb_92e7ebcb-9784-4538-98b1-5c599ce428e7/telemeter-client/0.log"
Sep 14 11:03:47.016993 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.016977    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c676b8b94-c2zxb_92e7ebcb-9784-4538-98b1-5c599ce428e7/reload/0.log"
Sep 14 11:03:47.040178 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.040154    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c676b8b94-c2zxb_92e7ebcb-9784-4538-98b1-5c599ce428e7/kube-rbac-proxy/0.log"
Sep 14 11:03:47.078464 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.078418    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c8cf6bc66-2q5hg_45892b23-fb26-4714-aafa-7be601377d6d/thanos-query/0.log"
Sep 14 11:03:47.100244 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.100230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c8cf6bc66-2q5hg_45892b23-fb26-4714-aafa-7be601377d6d/kube-rbac-proxy-web/0.log"
Sep 14 11:03:47.122005 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.121985    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c8cf6bc66-2q5hg_45892b23-fb26-4714-aafa-7be601377d6d/kube-rbac-proxy/0.log"
Sep 14 11:03:47.145661 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.145633    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c8cf6bc66-2q5hg_45892b23-fb26-4714-aafa-7be601377d6d/prom-label-proxy/0.log"
Sep 14 11:03:47.167999 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.167976    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c8cf6bc66-2q5hg_45892b23-fb26-4714-aafa-7be601377d6d/kube-rbac-proxy-rules/0.log"
Sep 14 11:03:47.191581 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:47.191555    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c8cf6bc66-2q5hg_45892b23-fb26-4714-aafa-7be601377d6d/kube-rbac-proxy-metrics/0.log"
Sep 14 11:03:49.928883 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.928846    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"]
Sep 14 11:03:49.929240 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.929189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc40abcd-8061-41b7-952e-597063da1130" containerName="maas-api"
Sep 14 11:03:49.929240 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.929200    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc40abcd-8061-41b7-952e-597063da1130" containerName="maas-api"
Sep 14 11:03:49.929310 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.929266    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc40abcd-8061-41b7-952e-597063da1130" containerName="maas-api"
Sep 14 11:03:49.932161 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.932139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:49.934255 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.934238    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qn42v\"/\"default-dockercfg-dgh7x\""
Sep 14 11:03:49.934922 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.934900    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qn42v\"/\"openshift-service-ca.crt\""
Sep 14 11:03:49.934993 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.934974    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qn42v\"/\"kube-root-ca.crt\""
Sep 14 11:03:49.949698 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:49.949676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"]
Sep 14 11:03:50.111820 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.111794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-lib-modules\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.111991 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.111827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqp7\" (UniqueName: \"kubernetes.io/projected/c5de2f39-1765-4156-8b32-41e662dfc063-kube-api-access-gpqp7\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.111991 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.111851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-podres\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.112118 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.111995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-proc\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.112118 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.112053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-sys\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213427 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-lib-modules\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213427 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqp7\" (UniqueName: \"kubernetes.io/projected/c5de2f39-1765-4156-8b32-41e662dfc063-kube-api-access-gpqp7\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213427 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-podres\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213683 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-proc\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213683 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-sys\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213683 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-lib-modules\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213683 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-sys\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213683 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-proc\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.213683 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.213625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5de2f39-1765-4156-8b32-41e662dfc063-podres\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.221066 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.221042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqp7\" (UniqueName: \"kubernetes.io/projected/c5de2f39-1765-4156-8b32-41e662dfc063-kube-api-access-gpqp7\") pod \"perf-node-gather-daemonset-nnqlq\" (UID: \"c5de2f39-1765-4156-8b32-41e662dfc063\") " pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.241803 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.241780    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:50.510424 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.510339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"]
Sep 14 11:03:50.513153 ip-10-0-133-125 kubenswrapper[2563]: W0914 11:03:50.513126    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc5de2f39_1765_4156_8b32_41e662dfc063.slice/crio-5e56ef0a2f40cc503300f2e436e8db3f43cb135e582f9290c8e4ced1e7d27604 WatchSource:0}: Error finding container 5e56ef0a2f40cc503300f2e436e8db3f43cb135e582f9290c8e4ced1e7d27604: Status 404 returned error can't find the container with id 5e56ef0a2f40cc503300f2e436e8db3f43cb135e582f9290c8e4ced1e7d27604
Sep 14 11:03:50.567461 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:50.567431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq" event={"ID":"c5de2f39-1765-4156-8b32-41e662dfc063","Type":"ContainerStarted","Data":"5e56ef0a2f40cc503300f2e436e8db3f43cb135e582f9290c8e4ced1e7d27604"}
Sep 14 11:03:51.573891 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:51.573852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq" event={"ID":"c5de2f39-1765-4156-8b32-41e662dfc063","Type":"ContainerStarted","Data":"3572cfe0cbf3b596fb5961c3d3433999952a6ba13b3601df5a016c91317275d9"}
Sep 14 11:03:51.574243 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:51.573980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:51.589082 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:51.589041    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq" podStartSLOduration=2.589021876 podStartE2EDuration="2.589021876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:51.588217642 +0000 UTC m=+2320.143405567" watchObservedRunningTime="2026-09-14 11:03:51.589021876 +0000 UTC m=+2320.144209802"
Sep 14 11:03:51.615423 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:51.615405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fqmh_329e2c19-8fd6-4b7c-8d99-3b8b9a336f41/dns/0.log"
Sep 14 11:03:51.635702 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:51.635680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fqmh_329e2c19-8fd6-4b7c-8d99-3b8b9a336f41/kube-rbac-proxy/0.log"
Sep 14 11:03:51.791983 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:51.791959    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5l8m9_e0aaadf1-3ea1-463d-b305-7fa7999dcfaf/dns-node-resolver/0.log"
Sep 14 11:03:52.418474 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:52.418445    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hwrdf_b0f1aa06-09e0-4f42-ad79-529290b9db2c/node-ca/0.log"
Sep 14 11:03:53.797956 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:53.797927    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb779b95-ww4qf_b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33/kube-auth-proxy/0.log"
Sep 14 11:03:54.857855 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:54.857826    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hmlhh_029e515a-26fc-4f35-b198-36597938ede6/serve-healthcheck-canary/0.log"
Sep 14 11:03:55.611730 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:55.611699    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sl6hx_3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1/kube-rbac-proxy/0.log"
Sep 14 11:03:55.646929 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:55.646902    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sl6hx_3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1/exporter/0.log"
Sep 14 11:03:55.682387 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:55.682363    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sl6hx_3001c318-bbe2-4f20-bc2a-1b7a84a5b0e1/extractor/0.log"
Sep 14 11:03:57.588678 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:57.588650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qn42v/perf-node-gather-daemonset-nnqlq"
Sep 14 11:03:58.837646 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:58.837614    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757dcccd8b-fsnj2_36389ec5-c492-4589-9823-53de660d9c77/manager/0.log"
Sep 14 11:03:58.860211 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:03:58.860187    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757dcccd8b-fsnj2_36389ec5-c492-4589-9823-53de660d9c77/copy-manifests/0.log"
Sep 14 11:04:00.514107 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:00.514030    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-594c4c57b7-t7hw4_fcf2aaad-ed53-43ff-8316-91b321fa72cf/manager/0.log"
Sep 14 11:04:00.558095 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:00.558070    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mgzhb_feba8fb8-7800-4069-9c0e-21be998399c2/openshift-lws-operator/0.log"
Sep 14 11:04:01.057025 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:01.056998    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb779b95-ww4qf_b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33/kube-auth-proxy/0.log"
Sep 14 11:04:03.279367 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:03.279289    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb779b95-ww4qf_b47b4f3b-e44c-4efa-bbc2-ec5ae7520c33/kube-auth-proxy/0.log"
Sep 14 11:04:07.236110 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.236067    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6xzkn_b820a1e5-95bd-4e5e-a5eb-b42b045ef0f2/kube-multus/0.log"
Sep 14 11:04:07.490138 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.490069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4hh7_a18d0c66-674c-45fc-8fbe-d283baeeb203/kube-multus-additional-cni-plugins/0.log"
Sep 14 11:04:07.520546 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.520525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4hh7_a18d0c66-674c-45fc-8fbe-d283baeeb203/egress-router-binary-copy/0.log"
Sep 14 11:04:07.545214 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.545190    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4hh7_a18d0c66-674c-45fc-8fbe-d283baeeb203/cni-plugins/0.log"
Sep 14 11:04:07.569499 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.569473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4hh7_a18d0c66-674c-45fc-8fbe-d283baeeb203/bond-cni-plugin/0.log"
Sep 14 11:04:07.599322 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.599287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4hh7_a18d0c66-674c-45fc-8fbe-d283baeeb203/routeoverride-cni/0.log"
Sep 14 11:04:07.625458 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.625436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4hh7_a18d0c66-674c-45fc-8fbe-d283baeeb203/whereabouts-cni-bincopy/0.log"
Sep 14 11:04:07.649454 ip-10-0-133-125 kubenswrapper[2563]: I0914 11:04:07.649428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4hh7_a18d0c66-674c-45fc-8fbe-d283baeeb203/whereabouts-cni/0.log"